MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Medical Informatics

Background:

• Multi-modal prompt learning tunes vision-language (V-L) models using text and image prompts.
• Current methods often neglect prompt dependencies and face challenges adapting to the medical domain due to data gaps.
• This limits the effectiveness of V-L models in specialized medical applications.

Purpose of the Study:

• To propose a Multi-modal Collaborative Prompt Learning (MCPL) pipeline for aligning medical text-image representations.
• To enhance the comprehension of medical reports and images by V-L models.
• To reduce the tuning costs for V-L models in medical downstream tasks.

Main Methods:

• Constructed an anatomy-pathology (AP) prompt incorporating instance-level medical information.
• Developed a graph-guided prompt collaboration module (GPCM) for multi-way prompt couplings.
• Implemented a novel prompt configuration scheme for improved interpretability within self-attention layers.

Main Results:

• MCPL demonstrated excellent effectiveness and generalization across medical classification and object detection datasets.
• The pipeline successfully aligned medical text-image representations for downstream tasks.
• Achieved superior performance compared to existing state-of-the-art prompt learning methods.

Conclusions:

• MCPL offers a reliable multi-modal prompt learning paradigm for the medical domain.
• The proposed method effectively addresses the gap between general and medical V-L model adaptation.
• MCPL significantly reduces the computational cost associated with tuning V-L models for medical applications.