Medical Vision-Language Modeling With Semantic Interaction and Adaptive Refinement Prompting for Bias Mitigation

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Computer Vision and Natural Language Processing

Background:

• Vision-Language Models (VLMs) show promise in medical report generation and VQA.
• Pixel-level medical image segmentation is underexplored but crucial for clinical applications.
• Medical datasets often exhibit biases, limiting VLM robustness and real-world applicability.

Purpose of the Study:

• To propose a unified medical vision-language model for diverse clinical tasks, including segmentation.
• To enhance pixel-level vision and language representation learning through a semantic interaction mechanism.
• To mitigate data bias effects using an adaptive refinement prompting method.

Main Methods:

• Developed a unified medical vision-language model.
• Incorporated a semantic interaction mechanism for improved pixel-level learning.
• Implemented an adaptive refinement prompting method to address data biases by iteratively re-prompting hard samples.

Main Results:

• The proposed model achieved superior performance in medical VQA and segmentation tasks.
• Experiments on eight datasets demonstrated the model's effectiveness compared to nine state-of-the-art methods.
• The approach shows potential for advancing real-world clinical applications of medical VLMs.

Conclusions:

• The unified medical VLM with semantic interaction and adaptive refinement prompting effectively handles diverse clinical tasks.
• The model demonstrates improved robustness and performance, particularly in segmentation, by addressing data biases.
• This work paves the way for more reliable and applicable VLMs in clinical settings.