Consistency-Queried Transformer for Audio-Visual Segmentation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Audio-visual segmentation (AVS) aims to segment objects within multimedia content by integrating audio and visual information.
• Existing AVS methods often struggle with multimodal inconsistencies, where visual features can overshadow audio cues, leading to suboptimal performance.
• Effective interaction between audio and visual features is crucial for advancing the multimodal domain.

Purpose of the Study:

• To propose a novel framework, the Consistency-Queried Transformer (CQFormer), to address multimodal inconsistencies in audio-visual segmentation.
• To enhance the fusion and interaction between audio and visual features for more accurate segmentation.
• To improve the accuracy and stability of object segmentation in audio-visual content.

Main Methods:

• The study introduces the Consistency-Queried Transformer (CQFormer), utilizing a transformer architecture.
• Key components include a Consistency Query Generator (CQG) and a Query-Aligned Matching (QAM) module.
• Noise Contrastive Estimation (NCE) loss is employed to minimize distributional differences between audio and visual features, promoting better modality matching and consistency.

Main Results:

• CQFormer demonstrated state-of-the-art performance on a popular audio-visual segmentation benchmark dataset.
• The framework effectively enhances modality matching and consistency through NCE loss.
• Incorporating consistency queries during decoding improved object-level semantic information and segmentation stability.

Conclusions:

• The proposed CQFormer framework effectively addresses multimodal inconsistencies in audio-visual segmentation.
• The novel approach leads to significant improvements in segmentation accuracy and stability.
• CQFormer represents a substantial advancement in audio-visual segmentation techniques.