DVPT: Dynamic Visual Prompt Tuning of large pre-trained models for medical image analysis

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Computer Vision

Background:

• Large pre-trained models offer rich representations for medical tasks.
• Current fine-tuning methods (full or layer-specific) overlook medical image variability, impacting efficiency and effectiveness.
• Parameter-efficient fine-tuning (PEFT) is crucial for adapting models to specialized domains.

Purpose of the Study:

• To explore parameter-efficient fine-tuning (PEFT) for medical image analysis.
• To introduce a novel method, Dynamic Visual Prompt Tuning (DVPT), for efficient knowledge extraction from large models.
• To enhance adaptation of pre-trained models to the medical domain and improve data efficiency.

Main Methods:

• Developed Dynamic Visual Prompt Tuning (DVPT), a novel PEFT method.
• DVPT utilizes a lightweight bottleneck layer to transform frozen features for domain-specific distribution learning.
• Learns sample-specific features via cross-attention between transformed features and dynamic visual prompts, shared across Transformer layers.

Main Results:

• DVPT efficiently adapts pre-trained models to medical imaging tasks.
• Achieved superior performance compared to state-of-the-art PEFT methods and full fine-tuning on medical classification and segmentation.
• Demonstrated significant improvements with minimal trainable parameters (0.5%), enhancing data efficiency and reducing storage costs.

Conclusions:

• DVPT offers an effective and efficient PEFT strategy for medical image analysis.
• The method significantly improves model performance and data efficiency, especially with limited labeled data.
• DVPT presents a promising approach for leveraging large pre-trained models in the medical field, reducing computational and storage overhead.