DICCR: Double-gated intervention and confounder causal reasoning for vision-language navigation

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Vision-language navigation (VLN) agents must correlate visual scenes with text instructions for sequential decision-making.
• Existing methods often overlook multi-modal data biases and spurious correlations between vision and text.
• Causality offers a framework to understand and mitigate these multi-modal relationships.

Purpose of the Study:

• To propose a novel vision-language navigation method, DICCR, that utilizes causal reasoning to address multi-modal biases.
• To weaken potential spurious correlations between visual and textual modalities through cross-modal causal reasoning.
• To enhance agent decision-making by reducing reliance on false associations.

Main Methods:

• Developed a causal graph of confounder factors using cross-modal reasoning for navigation.
• Employed front-door and back-door causal interventions guided by semantic relations to reduce biases.
• Designed a joint local-global causal attention module and a feature fusion matching algorithm (FFM).

Main Results:

• The DICCR model demonstrated significant improvements on benchmark datasets R2R, REVERIE, and RxR.
• Achieved a 3.25% increase in SPL and 4.13% in SR metrics on the R2R dataset.
• Outperformed baseline models, establishing a new state-of-the-art performance.

Conclusions:

• Causal reasoning effectively mitigates spurious correlations and biases in multi-modal data for VLN.
• The proposed DICCR model offers a robust and effective solution for vision-language navigation tasks.
• The findings highlight the importance of causal inference in developing more reliable AI agents.