Polyphonic training set synthesis improves self-supervised urban sound classification | JoVE Visualize

Area of Science:

Environmental acoustics
Machine learning
Signal processing

Background:

Machine listening systems require extensive expert annotations for training, which are often scarce.
Self-supervised learning (SSL) and training set synthesis are emerging paradigms to address data limitations in acoustic monitoring.
Previous research has largely explored SSL and synthesis independently, with limited understanding of their combined effects or data curation impacts.

Purpose of the Study:

To investigate a two-stage approach combining self-supervised learning and training set synthesis for environmental acoustic monitoring.
To evaluate the effectiveness of combining these paradigms for urban sound classification.
To determine the impact of data curation, specifically the geographical origin of acoustic events, on training set synthesis.

Main Methods:

A two-stage approach was implemented: first, a self-supervised pretext task (Audio2Vec skip-gram inpainting) on unlabeled spectrograms.
Second, a supervised downstream task of multilabel urban sound classification using synthetic scenes generated via training set synthesis.
The study analyzed the influence of geographical origin of acoustic events within the synthetic training data.

Main Results:

Training set synthesis demonstrated a greater overall performance improvement compared to self-supervised learning alone.
The geographical origin of the acoustic events used in training set synthesis was found to have a significant impact on classification performance.
Combining SSL and synthesis provided a viable strategy for enhancing acoustic event detection models.

Conclusions:

Training set synthesis is a highly effective method for improving machine listening systems in environmental acoustics, outperforming standalone SSL.
Careful consideration of the geographical provenance of data is crucial for optimizing training set synthesis.
The proposed two-stage methodology offers a promising direction for developing robust acoustic monitoring systems with limited labeled data.