Structure Causal Models and LLMs Integration in Medical Visual Question Answering.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Causal Inference

Background:

• Medical Visual Question Answering (Med-VQA) faces challenges due to inherent biases between medical images and questions.
• These cross-modal biases hinder the accurate inference of medically relevant answers from complex data.

Purpose of the Study:

• To develop a causal inference framework to eliminate confounding effects in Med-VQA.
• To enhance the precision and reliability of question-answering sessions in the medical domain.

Main Methods:

• Proposed a novel causal graph structure to model the interaction between visual and textual data.
• Utilized mutual information to identify spurious correlations and a multi-variable resampling front-door adjustment to mitigate confounding.
• Implemented a multi-form prompt strategy to improve model comprehension of complex medical data.

Main Results:

• The proposed causal framework significantly improved Med-VQA accuracy across three benchmark datasets.
• Demonstrated the method's ability to establish true causal correlations, overcoming challenges posed by complex medical data.

Conclusions:

• The causal inference framework effectively addresses cross-modal bias in Med-VQA.
• This approach leads to more accurate and causally sound medical question answering, advancing the field of AI in healthcare.