Interpretable Visual Question Answering by Reasoning on Dependency Trees.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Interpretable visual question answering (VQA) requires effective collaborative reasoning for image-question pairs.
• Current methods often rely on annotations or handcrafted rules, limiting performance and increasing workload.
• A need exists for models that align image and language domains in diverse, unrestricted scenarios.

Purpose of the Study:

• To propose a novel neural network model for interpretable VQA that performs global reasoning on question dependency trees.
• To enhance collaborative reasoning by integrating visual and linguistic information through a structured approach.
• To develop a VQA system capable of gradual, interpretable reasoning aligned with question structure.

Main Methods:

• A parse-tree-guided reasoning network (PTGRN) was developed, utilizing a dependency tree parsed from the question.
• The PTGRN incorporates three modules: an attention module for word-level visual evidence, a gated residual composition module for evidence aggregation, and a parse-tree-guided propagation module.
• This architecture facilitates question-driven reasoning by propagating mined evidence along the parse tree.

Main Results:

• Experiments on relational datasets demonstrated the superiority of PTGRN over current state-of-the-art VQA methods.
• The model achieved improved performance in compositional reasoning tasks.
• Visualization results highlighted the system's explainable reasoning capabilities.

Conclusions:

• The proposed PTGRN effectively addresses the underexplored area of collaborative reasoning in interpretable VQA.
• The model's parse-tree-guided approach enables gradual derivation of image cues, enhancing interpretability.
• PTGRN offers a promising direction for developing more robust and explainable VQA systems.