Knowledge-enhanced Parameter-efficient Transfer Learning with METER for medical vision-language tasks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Medical Informatics

Background:

• Full fine-tuning of pre-trained models for medical tasks is computationally expensive.
• Parameter-efficient fine-tuning (PEFT) methods reduce costs but can underperform due to domain gaps.
• Adapting general vision-language (VL) models to specialized medical domains remains challenging.

Purpose of the Study:

• To propose Knowledge-enhanced Parameter-efficient Transfer Learning with METER (KPL-METER) for medical VL tasks.
• To enhance model performance by integrating external medical knowledge into PEFT.
• To address the domain gap between general pre-trained models and medical applications.

Main Methods:

• Developed KPL-METER, combining PEFT with external domain-specific knowledge.
• Introduced Sharing Adapter (SAdapter) for multi-modal branches to maintain uni-modal features and cross-modal consistency.
• Implemented a novel knowledge extraction and parameter-free modeling strategy using the Unified Medical Language System (UMLS).
• Added Adapters to image and text encoders to further refine uni-modal features.

Main Results:

• KPL-METER outperformed other PEFT methods on medical VL tasks using fewer parameters.
• KPL-METER-MED, with medical-tailored encoders, achieved higher performance with fewer tuned parameters than previous medical domain models.
• The proposed methods effectively bridge the domain gap and improve performance.

Conclusions:

• KPL-METER architecture successfully adapts general VL models for medical applications.
• Knowledge extraction and fusion methods significantly enhance performance by incorporating medical domain-specific knowledge.
• The study provides an efficient and effective solution for medical VL tasks.