Enhancing Sound Source Localization via False Negative Elimination.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Sound source localization is crucial for understanding audio-visual scenes.
• Current methods often use contrastive learning but suffer from false negatives, where similar sounds are incorrectly excluded.
• This misalignment degrades audio-visual feature performance.

Purpose of the Study:

• To propose a novel audio-visual learning framework to overcome the limitations of existing methods.
• To improve sound source localization accuracy by addressing the false negative issue.
• To enhance the alignment between audio and visual features.

Main Methods:

• Introduced a framework with two learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning (SACL).
• SSPL utilizes positive pairs for feature discovery and a predictive coding module, acting as a negative-free approach.
• SACL refines visual features and negatives, offering an alternative contrastive learning strategy.

Main Results:

• The proposed framework significantly outperforms state-of-the-art methods in sound source localization.
• Experiments demonstrate the effectiveness of both SSPL and SACL in improving audio-visual feature representation.
• The learned representations show versatility when applied to audio-visual event classification and object detection.

Conclusions:

• The novel audio-visual learning framework effectively addresses the false negative problem in sound source localization.
• The proposed SSPL and SACL schemes provide robust methods for audio-visual feature alignment and learning.
• The approach demonstrates broad applicability and improved performance across related audio-visual tasks.