Multimodal Neural Acoustic Fields for Immersive Mixed Reality

Area of Science:

• Computer Vision
• Acoustics
• Machine Learning

Background:

• Neural radiance fields (NeRFs) excel at synthesizing novel views of scenes from images.
• Synthesizing realistic spatial audio, especially in novel environments, remains a challenge.

Purpose of the Study:

• To introduce multimodal neural acoustic fields for synthesizing spatial sound.
• To enable immersive auditory experiences from novel viewpoints and in unseen environments.
• To enhance presence in augmented and virtual reality applications.

Main Methods:

• Developed a neural network model mapping environment geometry and visual features to audio characteristics.
• Introduced a hybrid transformer-convolutional neural network for capturing reverberation and generating spatial sound.
• Utilized audio-visual data from sparse positions to represent spatial acoustics.

Main Results:

• The method successfully produces nonlinear acoustic effects like reverberations.
• Demonstrated improved spatial audio quality compared to existing methods on synthetic and real-world data.
• Validated enhanced audio perception in immersive mixed reality applications through user studies.

Conclusions:

• Multimodal neural acoustic fields offer a novel approach to synthesizing realistic spatial audio.
• The proposed method effectively learns and generates spatial acoustics for immersive experiences.
• This technology has significant potential for advancing augmented and virtual reality applications.