Contrastive Positive Sample Propagation Along the Audio-Visual Event Line.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Natural environments contain coexisting visual and audio signals forming audio-visual events (AVEs).
• Accurate localization and categorization of AVEs in videos require discriminative feature learning.
• Existing methods often rely on audio-visual feature fusion, which can be suboptimal.

Purpose of the Study:

• To propose a new method, Contrastive Positive Sample Propagation (CPSP), for improved deep feature representation learning in AVE localization.
• To leverage full or weak labels as priors to construct accurate positive-negative samples for contrastive learning.
• To enhance the localization and categorization of audio-visual events in video segments.

Main Methods:

• Introduced the Contrastive Positive Sample Propagation (CPSP) method for audio-visual event localization.
• Developed comprehensive contrastive constraints including pair-level positive sample propagation (PSP) and segment/video-level positive sample activation (PSA_S, PSA_V).
• Proposed three novel contrastive objectives and applied them to fully and weakly supervised AVE localization, also exploring self-supervised positive sample propagation (SSPSP).

Main Results:

• The CPSP method yields refined audio-visual features that are more distinguishable from negative samples.
• These refined features significantly benefit classifier prediction accuracy for AVE localization.
• Experiments on AVE and VGGSound-AVEL100k datasets demonstrated the effectiveness and generalization of CPSP.

Conclusions:

• The proposed CPSP method offers a superior approach to deep feature representation learning for audio-visual event localization compared to traditional fusion methods.
• CPSP effectively utilizes label information to improve the contrastive learning process, leading to better localization and categorization of AVEs.
• The method shows strong performance and generalization capabilities on benchmark datasets.