Visual explainable artificial intelligence for graph-based visual question answering and scene graph curation

Area of Science:

• Artificial Intelligence
• Computer Vision
• Data Visualization

Background:

• Graph-based visual question answering (VQA) systems often lack transparency in their decision-making processes.
• Identifying and correcting errors in VQA models is crucial for improving performance and data quality.

Purpose of the Study:

• To develop a novel visualization approach for explainable AI (XAI) in graph-based VQA.
• To enable users to identify false predictions and directly correct model mistakes in the input space.
• To facilitate dataset curation and enhance the understanding of graph neural network (GNN) internal states.

Main Methods:

• The study proposes a visualization tool integrated with a GraphVQA framework.
• The system utilizes graph neural networks (GNNs) for VQA tasks, trained on the GQA dataset.
• The approach highlights internal GNN states to explain model predictions.

Main Results:

• The developed tool effectively supports users in identifying incorrect predictions and diagnosing underlying issues.
• A user study with domain experts validated the tool's utility and effectiveness.
• Quantitative measures and use-case demonstrations confirmed the system's capabilities.

Conclusions:

• The novel visualization approach significantly enhances explainability for graph-based VQA systems.
• The tool facilitates dataset curation by enabling direct error correction.
• The method is extensible to other graph-based question answering models.