A visual question answering method based on task decomposition

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Visual Question Answering (VQA) integrates computer vision and natural language processing to assess visual reasoning.
• Traditional VQA methods often rely on multimodal fusion, which can be prone to data bias and lack interpretability.
• Task decomposition VQA methods offer better interpretability and reduced bias but struggle with flexible natural language parsing.

Purpose of the Study:

• To improve the accuracy of task decomposition in VQA by leveraging semantic structural information from natural language.
• To enhance the reasoning execution performance in VQA systems.
• To develop a novel VQA model that overcomes limitations of existing task decomposition approaches.

Main Methods:

• Propose a Graph-to-Sequence Task Decomposition Network (Graph2Seq-TDN) utilizing semantic graph structures for natural language parsing.
• Implement a novel reasoning executor to augment symbolic reasoning capabilities.
• Validate the model on diverse datasets including CLEVR, CLEVR-Human, CLEVR-CoGenT, and GQA.

Main Results:

• The Graph2Seq-TDN model demonstrated superior answering accuracy compared to baseline methods.
• Program accuracy was significantly improved by the proposed task decomposition approach.
• The model achieved better performance with reduced training costs under equivalent accuracy.

Conclusions:

• Task decomposition guided by semantic structural information offers a promising direction for VQA.
• The Graph2Seq-TDN model effectively addresses challenges in natural language parsing for VQA task decomposition.
• The proposed reasoning executor enhances overall VQA performance and efficiency.