What's this about?
OmniEcho combines spatial audio, visual observations and language so an embodied agent can identify sound direction and navigate toward it.
What exactly is ambisonic audio?
Instead of recording only loudness over time, first-order ambisonics encodes the three-dimensional direction of a sound field, giving software clues similar to how two ears help a person turn toward a noise.
Why does this matter?
author-reported preprint; 22 HF upvotes.
- The benchmark covers 197 real scenes, 2,972 questions and 900 navigation samples.
- A rendering pipeline creates scalable training data while preserving geometry.
- Fine-grained direction and distance estimation remain open weaknesses.