Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models.

Area of Science:

• Medical AI
• Computer Vision
• Natural Language Processing

Background:

• Cross-modal contrastive learning (CLIP) shows promise in medical vision-language tasks but has suboptimal visual representations.
• Multimodal masked modeling excels in visual representation but struggles with direct cross-modal matching.

Purpose of the Study:

• To develop an efficient medical vision-language alignment method.
• To improve performance in medical image-text matching tasks such as retrieval and zero-shot classification.

Main Methods:

• Proposed ALTA (ALign Through Adapting), an efficient method adapting pretrained vision models from masked modeling.
• Utilized ~8% trainable parameters and <1/5 computational cost of masked record modeling.
• Integrated temporal-multiview radiograph inputs for enhanced information consistency.

Main Results:

• ALTA achieved superior performance in vision-language matching tasks.
• Outperformed counterparts by >4% in text-to-image accuracy and ~6% in image-to-text retrieval accuracy.
• Demonstrated improved vision and language understanding through efficient alignment.

Conclusions:

• ALTA offers an efficient and effective approach for medical vision-language alignment.
• The method enhances performance on critical medical AI tasks.
• Adaptation of pretrained models is key to overcoming limitations of existing methods.