Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) systems often rely on spurious correlations between language and vision, hindering generalization.
• Existing de-biasing methods address one modality at a time, often increasing bias in the other.

Purpose of the Study:

• To model and mitigate the confounding effect of vision and language bias simultaneously in VQA.
• To improve the generalization capability of VQA models by reducing dual-modality bias.

Main Methods:

• Proposed a counterfactual inference strategy to remove the influence of confounding effects.
• Leveraged causal explain-away relations to address vision and language bias concurrently.
• Developed a model trained to reduce both vision and language bias efficiently.

Main Results:

• The proposed method concurrently and efficiently reduces vision and language bias.
• Achieved improved accuracy on questions with numerical answers, an open problem in VQA.
• Outperformed state-of-the-art methods on the VQA-CP v2 datasets.

Conclusions:

• This work is the first to reduce biases from confounding effects of vision and language in VQA using causal explain-away relations.
• The developed strategy enhances VQA model generalization by tackling dual-modality bias.
• The method shows significant improvements, particularly for numerical reasoning tasks in VQA.