Medical Knowledge Intervention Prompt Tuning for Medical Image Classification

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Vision-language foundation models (VLMs) offer strong generalization in medical tasks.
• Full fine-tuning of VLMs is computationally expensive.
• Existing prompt tuning methods struggle to capture specific medical concepts for accurate classification.

Purpose of the Study:

• To develop a resource-efficient prompt tuning method for medical image classification.
• To leverage large language models (LLMs) for specialized medical knowledge integration.
• To improve the ability of VLMs to distinguish disease-specific features across medical imaging modalities.

Main Methods:

• Introducing Conditional Intervention of Large Language Models for Prompt Tuning (CILMP).
• Extracting disease-specific representations from LLMs.
• Intervening in a low-rank linear subspace to create disease-specific prompts.
• Incorporating a conditional mechanism for instance-adaptive prompt generation.

Main Results:

• CILMP effectively transfers specialized medical knowledge from LLMs to VLMs.
• The method generates instance-adaptive prompts tailored to individual medical images.
• Extensive experiments show CILMP outperforms existing state-of-the-art prompt tuning techniques.
• Demonstrated consistent improvements across diverse medical image datasets.

Conclusions:

• CILMP offers an efficient and effective approach for medical image classification using prompt tuning.
• Integrating LLM knowledge enhances VLM adaptability and performance in specialized medical domains.
• The proposed method addresses limitations of current prompt tuning in capturing nuanced medical concepts.