Disease-Informed Adaptation of Vision-Language Models

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Computer Vision

Background:

• Developing AI foundation models for medical imaging is limited by expertise scarcity and high data annotation costs.
• Existing transfer learning methods struggle to adapt foundation models to new diseases with limited available data.
• Poor performance in AI models arises from difficulties in adapting to novel disease concepts not present in pre-training datasets.

Purpose of the Study:

• To propose a novel transfer learning method for adapting foundation Vision-Language Models (VLMs) to new diseases using minimal examples.
• To enable efficient adaptation of VLMs by learning nuanced representations of new disease concepts.
• To leverage the joint visual-linguistic capabilities of VLMs for improved few-shot learning in medical contexts.

Main Methods:

• Introduced a novel disease prototype learning framework.
• Developed disease-informed contextual prompting to capitalize on VLMs' joint visual-linguistic abilities.
• Evaluated the method across multiple pre-trained medical VLMs and various tasks.

Main Results:

• The proposed method enables VLMs to rapidly learn new disease concepts even with limited data.
• Demonstrated significant performance enhancements compared to existing adaptation techniques.
• Showcased the effectiveness across multiple medical VLMs and diverse downstream tasks.

Conclusions:

• The disease-informed contextual prompting framework offers an efficient solution for few-shot adaptation of VLMs in medical imaging.
• This approach addresses the challenge of limited data for new diseases, improving AI model performance.
• The method holds promise for accelerating the development and deployment of AI tools in clinical settings.