Joint Spatio-Temporal-Frequency Representation Learning for Improved Sound Event Localization and Detection.

Area of Science:

• Machine Listening
• Acoustic Signal Processing
• Artificial Intelligence

Background:

• Sound event localization and detection (SELD) is vital for machine listening.
• Current SELD methods often analyze spatial, temporal, and frequency domains separately, limiting real-world performance.
• Accurate sound event characterization requires integrated analysis across these domains.

Purpose of the Study:

• To propose a novel SELD method that jointly learns features across spatial, temporal, and frequency domains.
• To improve the accuracy and robustness of sound event localization and detection.
• To address the limitations of existing SELD approaches in handling complex acoustic environments.

Main Methods:

• Development of the Spatio-Temporal-Frequency Fusion Network (STFF-Net).
• Utilizing Enhanced-3D (E3D) residual blocks with 3D convolutions and attention mechanisms.
• Incorporating the multi-ACCDOA format to manage overlapping sound events.

Main Results:

• The proposed STFF-Net effectively captures intricate correlations across spatial, temporal, and frequency domains.
• Extensive experiments on benchmark datasets show significant performance improvements over state-of-the-art methods.
• The method demonstrates superior capability in sound event localization and detection tasks.

Conclusions:

• The STFF-Net offers a significant advancement in SELD by enabling joint spatio-temporal-frequency feature learning.
• This integrated approach leads to more accurate and robust sound event localization and detection.
• The proposed method sets a new benchmark for SELD performance.