Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) models often exhibit language bias, relying on spurious correlations rather than true multimodal understanding.
• This bias hinders model generalization and leads to decreased performance on VQA tasks.
• Existing methods struggle to effectively mitigate this inherent language bias.

Purpose of the Study:

• To propose a novel modality fusion collaborative de-biasing algorithm (CoD) to address language bias in VQA.
• To improve the generalization capabilities of VQA models by enabling them to fully leverage multimodal information.
• To enhance the prediction accuracy of VQA models by mitigating reliance on superficial correlations.

Main Methods:

• Developed a collaborative de-biasing algorithm (CoD) that treats bias as modality information neglect.
• Employed a collaborative training approach for mutual modeling between different modalities.
• Implemented efficient feature fusion to integrate information from both visual and textual modalities.

Main Results:

• Demonstrated the effectiveness of the CoD algorithm across multiple VQA datasets (VQA-CP v2, VQA v2, VQA-VS).
• Achieved improved performance and generalization compared to baseline models.
• A basic baseline model utilizing CoD achieved 60.14% accuracy on the VQA-CP v2 dataset.

Conclusions:

• The proposed CoD algorithm effectively addresses language bias in VQA models.
• Collaborative training and modality fusion are crucial for leveraging multimodal knowledge and improving VQA performance.
• The CoD approach offers a promising direction for developing more robust and generalizable VQA systems.