Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Current Visual Question Answering (VQA) models often rely on superficial linguistic patterns, leading to poor generalization on out-of-distribution datasets.
• Existing methods to reduce language bias, like auxiliary question-only models, are complex and do not guarantee visual explainability or question sensitivity.

Purpose of the Study:

• To propose a novel, model-agnostic strategy called Counterfactual Samples Synthesizing and Training (CSST) to enhance VQA model robustness.
• To improve VQA models' visual explainability and question sensitivity, ensuring they focus on relevant image regions and linguistic nuances.

Main Methods:

• CSST involves two stages: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST).
• CSS generates modified data by masking critical image objects or question words and assigning pseudo-answers.
• CST trains VQA models on original and counterfactual samples, using supervised contrastive loss and a specialized sample selection mechanism to distinguish subtle differences.

Main Results:

• CSST effectively forces VQA models to attend to all crucial objects and words.
• Models trained with CSST demonstrate significantly improved visual explainability and question sensitivity.
• Achieved state-of-the-art performance on multiple out-of-distribution benchmarks, including VQA-CP v2, VQA-CP v1, and GQA-OOD.

Conclusions:

• The proposed CSST strategy offers a powerful and versatile approach to mitigate language biases in VQA models.
• CSST enhances fundamental VQA capabilities, leading to more reliable and interpretable AI systems.
• This method sets a new benchmark for VQA performance, particularly in challenging out-of-distribution scenarios.