Binaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds

Area of Science:

• Computer Vision
• Machine Learning
• Acoustics

Background:

• Humans excel at scene understanding using visual and auditory cues, but machines primarily rely on visual data.
• Developing machine capabilities for sound-based scene understanding remains an underexplored area.

Purpose of the Study:

• To develop a machine learning approach for scene understanding using only binaural audio.
• To enable machines to predict semantic masks, motion, and depth maps of sound-making objects from audio.
• To create a new audio-visual dataset of street scenes for training and evaluation.

Main Methods:

• A novel sensor setup with eight binaural microphones and a 360° camera was used to record a new street scene dataset.
• A cross-modal distillation framework transferred knowledge from vision 'teacher' models to a sound 'student' model, enabling training without human annotations.
• An auxiliary task, Spatial Sound Super-Resolution, was introduced to enhance sound directional resolution.

Main Results:

• The proposed multi-tasking network achieved good performance across all four tasks: semantic mask prediction, motion estimation, depth mapping, and sound super-resolution.
• Jointly training the four tasks proved mutually beneficial, leading to the best overall performance.
• Microphone configuration (number and orientation) significantly impacts performance.
• Complementary features from standard spectrograms and classic signal processing pipelines enhance auditory perception.

Conclusions:

• The developed approach demonstrates the potential of purely audio-based scene understanding for machines.
• Multi-task learning and specialized audio processing techniques like Spatial Sound Super-Resolution are effective for improving auditory perception.
• The new dataset and framework facilitate further research in sound-based scene understanding.