Sound Event Localization and Detection Using Imbalanced Real and Synthetic Data via Multi-Generator

Area of Science:

• Acoustics and Signal Processing
• Machine Learning
• Artificial Intelligence

Background:

• Sound event localization and detection (SELD) is crucial for understanding acoustic environments.
• The Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Task 3 faces challenges with imbalanced real and synthetic data.
• Training models on imbalanced datasets can lead to biased performance, favoring the majority data class.

Purpose of the Study:

• To propose a novel SELD method capable of effectively utilizing imbalanced real and synthetic datasets.
• To address the challenge of data imbalance in SELD tasks, particularly in the context of DCASE 2022 Task 3.
• To enhance the performance of SELD systems by developing a robust training strategy and neural network architecture.

Main Methods:

• A multi-generator approach was developed to sample real and synthetic data at a specific rate within a single batch, mitigating data imbalance.
• The proposed method integrates a residual convolutional neural network (RCNN) with a transformer encoder for processing real spatial sound scenes.
• Data augmentation techniques, including SpecAugment and time-frequency masking, were applied to enhance the dataset.
• Ensemble models were created by selecting and combining the best-performing individual models based on various structures and hyperparameters.

Main Results:

• The proposed SELD method, utilizing the multi-generator strategy, demonstrated improved performance compared to the baseline model.
• Both the single model and the ensemble model achieved superior results, indicating the effectiveness of the proposed approach.
• The integration of RCNN and transformer encoder architectures contributed to enhanced sound event localization and detection capabilities.

Conclusions:

• The developed multi-generator strategy is effective in addressing data imbalance in SELD tasks.
• The proposed RCNN and transformer encoder-based neural network architecture yields state-of-the-art performance for SELD.
• The findings suggest a promising direction for improving SELD systems in real-world acoustic scenarios with limited real data.