Graph-Enhanced Visual Prompting for Pre-Trained Models Adaptation in Medical Imaging Classification

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Adapting Vision Transformers (ViTs) for medical imaging faces challenges due to limited data and annotations, impacting training and generalization.
• Visual prompt learning offers parameter-efficient domain adaptation but requires task-specific semantic guidance, which is often unavailable in clinical settings.
• There is a need for automated methods to extract reliable semantic cues from existing clinical data for effective prompt generation.

Purpose of the Study:

• To introduce Graph-Enhanced Visual Prompting (GEVP), a novel framework for generating semantically rich prompts in medical imaging.
• To integrate cross-modal graph learning into prompt generation for Vision Transformers (ViTs) in medical applications.
• To enable robust medical image analysis and disease classification, particularly in data-scarce environments.

Main Methods:

• GEVP models image patches and clinical report tokens as nodes in a graph structure.
• A graph neural network is employed to capture spatial and semantic relationships between these nodes.
• Generated prompts are injected into a frozen ViT backbone to guide attention to diagnostically relevant regions without extensive fine-tuning.

Main Results:

• GEVP demonstrated superior performance compared to existing prompt- and adapter-based methods on six public medical imaging datasets.
• The framework achieved up to a +9.65% improvement in F1 score on imbalanced tasks.
• GEVP showed enhanced capabilities in classifying unseen diseases, highlighting its generalization potential.

Conclusions:

• Graph-Enhanced Visual Prompting (GEVP) effectively overcomes data scarcity in medical imaging by leveraging cross-modal graph learning for prompt generation.
• The proposed method enables parameter-efficient adaptation of Vision Transformers (ViTs) for medical image analysis, improving diagnostic accuracy.
• GEVP offers a robust solution for both report-available and report-absent scenarios, paving the way for wider clinical adoption.