An effective spatial relational reasoning networks for visual question answering.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Visual Question Answering (VQA) models typically focus on visual semantic features, neglecting spatial relationships between objects.
• Existing VQA research primarily emphasizes attention mechanisms and multi-modal fusion.

Purpose of the Study:

• To propose an effective spatial relationship reasoning network model for VQA.
• To integrate visual object semantic reasoning with spatial relationship reasoning for fine-grained multi-modal fusion.
• To address limitations in current VQA models regarding spatial understanding.

Main Methods:

• Developed a novel spatial relationship reasoning network combining semantic and spatial reasoning.
• Designed a sparse attention encoder for contextual information in semantic reasoning.
• Employed a graph neural network attention mechanism for modeling spatial relationships.
• Introduced a compact self-attention (CSA) mechanism to reduce redundancy and improve efficiency.

Main Results:

• The proposed model demonstrates superior performance on VQA 2.0 and GQA datasets.
• Achieved an accuracy of 71.18% on the VQA 2.0 dataset.
• Achieved an accuracy of 57.59% on the GQA dataset.

Conclusions:

• The integrated approach of semantic and spatial reasoning significantly enhances VQA performance.
• The proposed model effectively handles complex spatial relationship reasoning questions.
• The compact self-attention mechanism contributes to improved overall model performance and efficiency.