Weakly-Supervised 3D Spatial Reasoning for Text-Based Visual Question Answering.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Text-based Visual Question Answering (TextVQA) requires understanding spatial relationships between scene text and objects.
• Current 2D-based methods struggle with fine-grained spatial reasoning, limiting interpretability and performance.

Purpose of the Study:

• To improve TextVQA by integrating 3D geometric information for more robust spatial reasoning.
• To enhance the model's ability to interpret and utilize spatial context between visual elements and text.

Main Methods:

• Introduction of 3D geometric information into the spatial reasoning process.
• Proposal of a relation prediction module for precise object localization.
• Design of a depth-aware attention calibration module to refine OCR token attention based on object context.

Main Results:

• Achieved state-of-the-art performance on TextVQA and ST-VQA datasets.
• Demonstrated significant performance gains of 5.7% and 12.1% on spatial reasoning questions in TextVQA and ST-VQA, respectively.
• Validated generalizability on the text-based image captioning task.

Conclusions:

• Integrating 3D geometric information offers a superior approach to spatial reasoning in TextVQA.
• The proposed modules effectively enhance the model's understanding of object-text spatial relationships.
• The method shows promise for advancing visual question answering and related multimodal AI tasks.