Counterfactual Dual-Bias VQA: A Multimodality Debias Learning for Robust Visual Question Answering

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) models often exhibit language bias, relying heavily on questions and neglecting image content.
• Existing debiasing methods may not fully capture the nuances of language bias in VQA.

Purpose of the Study:

• To propose a novel multimodality counterfactual dual-bias model to mitigate linguistic bias in VQA models.
• To enhance the interpretability and question sensitivity of VQA models.

Main Methods:

• Designed a shared-parameterized dual-bias model incorporating visual and question counterfactual samples.
• Froze target VQA model parameters during dual-bias model training using cross-entropy and KL divergence loss.
• Re-trained the target VQA model using pseudo-labels generated from the frozen dual-bias model and a margin loss.

Main Results:

• The proposed counterfactual dual-bias model demonstrated superior effectiveness on VQA-CP datasets.
• Analysis of unsatisfactory performance on the VQA v2 dataset was conducted.

Conclusions:

• The multimodality counterfactual dual-bias model effectively addresses linguistic bias in VQA.
• The approach enhances VQA model robustness and accuracy by emphasizing relevant visual and textual information.