Visual question answering based on local-scene-aware referring expression generation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Current visual question answering (VQA) methods often rely on limited object categories and relationships, struggling with complex scenes.
• Existing VQA models exhibit insufficient decision-explanation capabilities due to a narrow focus on visual concepts.

Purpose of the Study:

• To enhance VQA by integrating rich, structurally unconstrained text expressions generated for images.
• To improve the representation of complex scenes and the accuracy of decision-making in VQA systems.

Main Methods:

• Proposed a novel method utilizing image-generated text expressions to provide richer image descriptions.
• Developed a joint-embedding multi-head attention network to model and co-attend visual features, question embeddings, and generated text expressions.
• Evaluated the method on the VQA v2 dataset for answer prediction and on RefCOCO datasets for expression quality.

Main Results:

• The proposed method significantly outperformed state-of-the-art methods in quantitative and qualitative VQA evaluations.
• Generated text expressions were evaluated and found to be effective in enhancing image descriptions.
• The joint-embedding network successfully modeled multiple information modalities for improved VQA performance.

Conclusions:

• Integrating rich text expressions with visual and textual information is effective for advancing visual question answering.
• The proposed joint-embedding multi-head attention network provides a robust framework for multimodal VQA.
• The approach demonstrates superior performance in both predicting answers and describing image content.