EXPLORING SELF-SUPERVISED CONTRASTIVE LEARNING OF SPATIAL SOUND EVENT REPRESENTATION.

Area of Science:

• Audio Signal Processing
• Machine Learning
• Artificial Intelligence

Background:

• Spatial audio analysis requires understanding both sound events and their locations.
• Unsupervised learning methods are crucial for leveraging large amounts of unlabeled audio data.
• Existing methods may not effectively capture joint spectral and spatial characteristics of audio.

Purpose of the Study:

• To develop a multi-channel framework for contrastive learning (MC-SimCLR) to encode 'what' and 'where' in spatial audio.
• To learn joint spectral and spatial representations from unlabeled spatial audio data.
• To enhance downstream tasks like event classification and sound localization.

Main Methods:

• Proposed a multi-level data augmentation pipeline augmenting waveforms, Mel spectrograms, and generalized cross-correlation (GCC) features.
• Introduced channel-wise augmentation including microphone order swapping and Mel/GCC channel masking.
• Utilized contrastive learning to learn joint spectral and spatial representations from unlabeled spatial audio.

Main Results:

• Linear layers on learned representations significantly outperformed supervised models in event classification accuracy.
• The framework achieved superior performance in sound localization, reducing localization error.
• Analysis confirmed the effectiveness of individual augmentation methods and fine-tuning strategies.

Conclusions:

• MC-SimCLR effectively learns joint spectral and spatial representations for spatial audio.
• The proposed augmentation strategies are crucial for the framework's success.
• This unsupervised approach offers a powerful alternative to supervised methods for spatial audio tasks.