Weakly-Supervised Video Object Grounding via Causal Intervention.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Weakly-supervised video object grounding (WSVOG) aims to localize objects described in text within videos using only video-sentence annotations.
• Existing WSVOG methods struggle with spurious associations, where non-object-related visual cues are incorrectly linked to text descriptions due to weak supervision and observational bias.
• This limitation hinders accurate and robust performance in pattern analysis and machine learning applications.

Purpose of the Study:

• To develop a unified causal framework to learn deconfounded object-relevant associations for WSVOG.
• To address the ambiguity and confounding effects inherent in current WSVOG approaches.
• To enhance the accuracy and robustness of video object grounding.

Main Methods:

• Designed a unified causal framework employing causal intervention based on the video data generation process.
• Proposed a novel spatial-temporal adversarial contrastive learning paradigm to handle the lack of fine-grained supervision for intervention.
• Implemented causal intervention via backdoor adjustment to mitigate confounding effects and learn true causality.

Main Results:

• The proposed framework successfully learns deconfounded object-relevant associations in an end-to-end manner.
• Experiments on IID and OOD datasets across three benchmarks demonstrate superior grounding performance compared to state-of-the-art methods.
• The approach shows significant improvements in accuracy and robustness.

Conclusions:

• The unified causal framework effectively overcomes spurious associations in WSVOG.
• The method provides a robust solution for accurate object localization in videos under weak supervision.
• This work advances the field of video understanding and pattern analysis.