M3AE-Distill: An Efficient Distilled Model for Medical Vision-Language Downstream Tasks

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision

Background:

• Multi-modal masked autoencoders (M3AE) are powerful medical vision-language (VL) models but suffer from large parameter sizes, hindering clinical deployment.
• Knowledge distillation (KD) effectively compresses uni-modal models, but its application to pre-training medical VL backbone models is not well-established.

Purpose of the Study:

• To develop M3AE-Distill, a lightweight medical VL model that maintains high performance while improving efficiency through knowledge distillation.
• To address the deployment challenges of large M3AE models in real-world clinical settings.

Main Methods:

• Employed hidden state and attention map distillation to guide the student model during pre-training.
• Introduced an attention-guided masking strategy to improve fine-grained image-text alignment.
• Developed two student variants, M3AE-Distill-Small and M3AE-Distill-Base, for flexible accuracy-efficiency trade-offs.

Main Results:

• M3AE-Distill demonstrated effectiveness across five medical VL datasets and three tasks.
• M3AE-Distill-Base achieved performance comparable to the teacher model.
• Achieved significant speedups: 2.11× during inference and 2.61× during fine-tuning.

Conclusions:

• M3AE-Distill offers an efficient and high-performing solution for medical vision-language tasks.
• The proposed distillation and masking strategies enhance model efficiency without sacrificing accuracy.
• The model variants provide practical options for deploying efficient medical VL systems.