Bridging the Semantic Gap in Medical Visual Question Answering With Prompt Learning

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Medical Visual Question Answering (Med-VQA) is vital for healthcare diagnostics and education.
• Data scarcity due to annotation costs hinders Med-VQA progress.
• Existing pre-training methods struggle to bridge the semantic gap for specific Med-VQA tasks.

Purpose of the Study:

• To introduce the Dynamic Semantic-Adaptive Prompting (DSAP) framework to enhance Med-VQA model performance.
• To address the semantic gap and improve accuracy in Med-VQA systems.
• To leverage prompt learning for more effective medical image analysis.

Main Methods:

• Developed the DSAP framework incorporating two prompting strategies: Semantic Alignment Prompting (SAP) and Dynamic Question-Aware Prompting (DQAP).
• SAP aligns multi-modal inputs with domain-specific contexts during fine-tuning.
• DQAP utilizes grammatical relationships between questions and answers to improve answer selection.

Main Results:

• DSAP framework pre-trained on ROCO, MedICaT, and MIMIC-CXR datasets.
• Evaluated against 15 existing Med-VQA models on VQA-RAD, SLAKE, and PathVQA datasets.
• Achieved a 1.9% average performance enhancement across benchmarks, demonstrating substantial improvement.

Conclusions:

• DSAP effectively addresses critical challenges in Med-VQA, particularly data scarcity and semantic gap issues.
• The proposed prompting strategies significantly enhance model accuracy and relevance.
• DSAP offers a promising direction for future advancements in medical artificial intelligence and AI-assisted diagnostics.