Unsupervised Synthetic Acoustic Image Generation for Audio-Visual Scene Understanding

Area of Science:

• Computer Vision
• Audio Signal Processing
• Machine Learning

Background:

• Acoustic images offer rich spatial audio information for scene understanding.
• Traditional acoustic image generation requires costly and complex microphone arrays.
• Single-microphone setups are more accessible but lack spatial audio capabilities.

Purpose of the Study:

• To develop methods for generating acoustic images from single-microphone cameras.
• To evaluate the effectiveness of these generated acoustic images in audio-visual scene understanding tasks.
• To compare the performance of different generative model architectures.

Main Methods:

• Proposed three generative models (Variational Autoencoder, U-Net, adversarial) trained on video and monaural audio.
• Models conditioned on video sequences and single-channel audio to produce spatialized audio.
• Trained models using microphone array data as ground truth to mimic array output.

Main Results:

• Generated acoustic images achieved notable performance in classification, cross-modal retrieval, and sound localization.
• The generated data performed comparably to real acoustic images from microphone arrays.
• Models demonstrated effectiveness across multimodal datasets and monaural audio/video datasets.

Conclusions:

• Single-microphone cameras can be leveraged to generate high-quality acoustic images.
• The proposed generative models offer a cost-effective alternative for audio-visual scene understanding.
• This approach significantly enhances the utility of readily available audio-visual data.