Semi-Supervised VQA Multi-Modal Explanation via Self-Critical Learning

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) explanation aims to make VQA model decisions human-understandable.
• Current methods often rely on single modalities (visual or textual), leading to ambiguity and logical inconsistencies.
• Collecting human-annotated explanations is costly and time-consuming.

Purpose of the Study:

• To develop a novel method for generating accurate and logically consistent VQA explanations.
• To address the limitations of uni-modal paradigms and reduce the reliance on expensive human annotations.
• To improve the interpretability of VQA models through multi-modal explanations.

Main Methods:

• Introduced a Semi-supervised VQA Multi-modal Explanation (SME) method.
• Employed self-critical learning to enhance logical consistency between answers and explanations using reward scores.
• Leveraged semi-supervised learning to utilize large datasets without human-annotated explanations.

Main Results:

• The SME method effectively generates comprehensive explanations by integrating visual and textual information.
• Demonstrated improved logical consistency between VQA model answers and their generated explanations.
• Achieved state-of-the-art performance on three VQA explanation datasets.

Conclusions:

• The proposed SME method offers a more effective and efficient approach to VQA explanation.
• Multi-modal explanations and self-critical learning significantly improve interpretability and consistency.
• The method's ability to leverage unlabeled data reduces annotation costs and enhances scalability.