Learning contrastive semantic decomposition for visual grounding

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Visual grounding links natural language descriptions to image regions.
• Current methods use separate encoders, potentially missing shared attributes and causing redundant fusion.

Purpose of the Study:

• To propose a novel Contrastive Semantic Decomposition network for Visual Grounding (CSDVG).
• To effectively decompose shared-specific semantic features and model cross-modality features for improved visual grounding.

Main Methods:

• Developed CSDVG with an associated semantic branch for shared features and an independent semantic branch for specific features.
• Introduced a relevance-driven loss function to balance shared and specific feature learning.

Main Results:

• CSDVG demonstrated superior performance compared to existing approaches.
• Experiments showed effectiveness across all tested datasets.

Conclusions:

• The proposed CSDVG effectively decomposes and models cross-modality features.
• CSDVG addresses limitations of independent encoders and redundant fusion in visual grounding tasks.