Vision-Language Model for Visual Question Answering in Medical Imagery

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Medical images are crucial in clinical diagnosis.
• Medical Visual Question Answering (VQA) systems can enhance diagnostic accuracy.
• Current VQA technology for medical applications is underdeveloped.

Purpose of the Study:

• To introduce an advanced transformer encoder-decoder architecture for medical VQA.
• To improve the performance of VQA systems in analyzing radiology images.
• To bridge the gap between current VQA capabilities and practical clinical application.

Main Methods:

• Image features extracted using Vision Transformer (ViT).
• Questions embedded using a textual encoder transformer.
• Concatenated visual and textual representations fed into a multi-modal decoder.
• Answer generation using an autoregressive approach.

Main Results:

• The proposed model was validated on VQA-RAD and PathVQA datasets.
• Achieved 84.99% closed and 72.97% open accuracy on VQA-RAD.
• Achieved 83.86% closed and 62.37% open accuracy on PathVQA.
• Reported BLUE scores indicate good alignment between predicted and true answers.

Conclusions:

• The transformer-based VQA model demonstrates strong performance on medical image datasets.
• The approach shows significant potential for improving diagnostic support in healthcare.
• Further development of this VQA system could lead to practical clinical tools.