Inverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) models often exploit question cues and answer biases rather than truly understanding images.
• Current VQA benchmarks may not fully capture the depth of visuo-linguistic understanding required.

Purpose of the Study:

• Introduce the inverse problem of VQA (iVQA) as a novel benchmark for AI.
• Develop a variational iVQA model for generating diverse and relevant questions.
• Propose a reinforcement learning framework using iVQA to diagnose VQA model weaknesses.

Main Methods:

• Proposed a variational iVQA model to generate questions from image-answer pairs.
• Utilized iVQA within a reinforcement learning framework to probe VQA model beliefs.
• Analyzed the 'belief sets' of existing VQA models to identify flaws.

Main Results:

• The proposed iVQA model generates diverse, grammatically correct, and content-correlated questions.
• iVQA serves as a more challenging benchmark, requiring deeper image comprehension than VQA.
• Diagnosed existing VQA models, revealing significant erroneous beliefs and intrinsic weaknesses.

Conclusions:

• iVQA is a valuable tool for advancing visuo-linguistic understanding in AI.
• The iVQA-based diagnostic framework offers new insights into VQA model limitations.
• Findings suggest avenues for improving the robustness and accuracy of future VQA systems.