Embedding-based pair generation for contrastive representation learning in audio-visual surveillance data.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Smart cities utilize sensors like microphones and RGB cameras for citizen safety and comfort.
• Self-supervised learning, particularly contrastive learning, is crucial for analyzing audio-visual data due to high annotation costs.
• Existing audio-visual contrastive learning methods face challenges with false negatives and information bottlenecks.

Purpose of the Study:

• To address limitations in audio-visual contrastive learning for smart city surveillance.
• To propose a novel method for generating more effective contrastive pairs.
• To improve the learning of audio-visual representations for downstream tasks.

Main Methods:

• Developed a novel method for generating contrastive pairs based on embedding distances between modalities, moving beyond solely temporal cues.
• Introduced a new loss function for multiple positives.
• Experimentally validated the approach on real-world surveillance data.

Main Results:

• The proposed method effectively generates semantically synchronized pairs, easing the information bottleneck.
• Learned representations are applicable to diverse downstream tasks like event localization, anomaly detection, and event search.
• Achieved performance comparable to state-of-the-art specialized methods.

Conclusions:

• The novel contrastive learning approach enhances audio-visual representation learning for smart city applications.
• The method effectively handles challenges like false negatives and information bottlenecks in surveillance data.
• The learned representations offer a versatile foundation for multiple critical smart city monitoring tasks.