Interpretable medical image Visual Question Answering via multi-modal relationship graph learning.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical Visual Question Answering (VQA) is crucial for multi-modal large language models (LLMs) in healthcare.
• Existing medical VQA datasets are limited in size and question complexity, lacking clinical reasoning capabilities.
• Previous rule-based VQA approaches exhibited high error rates in label extraction.

Purpose of the Study:

• To address the limitations of current medical VQA datasets and methods.
• To develop a large-scale, clinically relevant VQA dataset focused on chest X-ray images.
• To propose a novel VQA method that enhances reasoning and faithfulness in medical applications.

Main Methods:

• Developed a large-scale medical VQA dataset (Medical-CXR-VQA) using LLMs for chest X-ray analysis.
• Trained an LLM to improve label extraction accuracy by 62% compared to rule-based methods.
• Proposed a novel VQA method utilizing spatial, semantic, and implicit relationship graphs with graph attention for logical reasoning.

Main Results:

• The proposed graph-based VQA method effectively learns logical reasoning paths.
• The Medical-CXR-VQA dataset contains detailed questions about abnormalities, locations, and types in chest X-rays.
• The approach demonstrates evidence and faithfulness, critical qualities for clinical deployment.

Conclusions:

• The Medical-CXR-VQA dataset and the novel graph-based VQA method advance medical VQA capabilities.
• This work provides a foundation for fine-tuning and training more sophisticated multi-modal LLMs in medicine.
• The developed reasoning paths can be integrated into LLM prompt engineering and chain-of-thought processes for enhanced clinical decision support.