Bridging the Modality Gap in Medical Vision-Language Models: A Hybrid Contrastive-Optimal Transport Framework for...

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Machine Learning

Background:

• Vision-language models (VLMs) in healthcare struggle with a modality gap, where image and text data embeddings are poorly aligned.
• This misalignment hinders cross-modal understanding and limits the performance of downstream AI tasks in medicine.
• Current solutions often overlook the geometric constraints inherent in multimodal contrastive learning.

Purpose of the Study:

• To develop a novel framework for medical modality alignment that addresses both instance-level and distribution-level gaps.
• To improve the performance of vision-language models in healthcare by reducing the modality gap.
• To create diagnosis-aware embeddings compatible with existing clinical workflows.

Main Methods:

• A novel framework combining contrastive learning and entropy-regularized optimal transport for medical modality alignment.
• A medical condition-driven association strategy for defining positive pairs based on shared pathologies.
• An intra-modality negative sampling scheme to prevent cross-modal separation and a lightweight embedding refinement network.

Main Results:

• Significant reduction in the modality gap, evidenced by increased alignment scores (0.33-0.73).
• Improved retrieval precision (22%-33%) and zero-shot classification accuracy (13%-48%).
• A 4.27-fold reduction in clustering dispersion metrics on standard medical benchmarks.

Conclusions:

• The proposed framework effectively bridges the modality gap in medical vision-language models.
• The approach enhances cross-modal understanding and downstream task performance in healthcare AI.
• Diagnosis-aware embeddings are generated, ensuring compatibility with clinical pipelines.