Knowledge-enhanced medical image classification via descriptive priors from large language models.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Current medical image classification relies heavily on visual features, often missing crucial diagnostic details.
• There is a need for approaches that incorporate medical expertise to enhance diagnostic accuracy.

Purpose of the Study:

• To propose a novel knowledge-enhanced model (KEM) for medical image classification.
• To leverage medical large vision-language models (Medical LVLMs) as domain experts to generate descriptive priors.
• To improve the recognition of subtle disease patterns by integrating textual and visual information.

Main Methods:

• KEM prompts Medical LVLMs to generate multi-dimension clinical descriptions for input images.
• Descriptive priors are encoded and fused with visual features using a dual cross-attention module.
• This module facilitates bidirectional interaction and alignment between textual and visual modalities.

Main Results:

• The proposed KEM method significantly outperforms state-of-the-art vision-only models.
• KEM demonstrates strong generalization capabilities across diverse clinical settings.
• Experiments were conducted on four benchmark datasets to validate the model's performance.

Conclusions:

• KEM effectively integrates medical expertise into image classification through descriptive priors.
• The model enhances the recognition of subtle disease patterns by aligning textual and visual cues.
• KEM offers a promising approach for more accurate and reliable medical image diagnosis.