Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Contrastive Language-Image Pre-training (CLIP) shows promise in medical imaging but requires extensive data and computational power.
• Current CLIP applications are limited to data-rich modalities like chest X-rays, neglecting under-explored areas such as mammography.
• Mammography presents unique challenges including scarce labeled data, high-resolution images with small regions of interest, and class imbalance.

Purpose of the Study:

• To adapt the full CLIP model for mammography analysis, overcoming existing data and computational constraints.
• To develop novel techniques for leveraging multi-view mammography data and enhancing focus on detailed features.
• To address data limitations through parameter-efficient fine-tuning of medical knowledge-pretrained large language models.

Main Methods:

• Developed a specialized supervision framework utilizing the multi-view nature of mammography.
• Designed a symmetric local alignment module to improve focus on high-resolution image details.
• Incorporated parameter-efficient fine-tuning for large language models with medical pre-training.

Main Results:

• The proposed multi-view and multi-scale alignment (MaMA) method demonstrated superior performance across three distinct tasks.
• Evaluated on the large-scale EMBED and RSNA-Mammo mammography datasets, MaMA outperformed state-of-the-art baselines.
• Achieved comparable results with a significantly reduced model size (52% of the largest baseline).

Conclusions:

• The MaMA method offers an effective and efficient adaptation of CLIP for mammography analysis.
• This approach successfully tackles challenges like data scarcity and image complexity in mammography.
• The findings suggest a promising direction for advancing AI in under-explored medical imaging modalities.