Bridged Semantic Alignment for Zero-Shot 3D Medical Image Diagnosis

Area of Science:

• Medical imaging analysis
• Artificial intelligence in healthcare
• Computer vision and natural language processing

Background:

• Supervised learning for medical diagnosis requires extensive manual annotations, limiting data availability and abnormality diversity.
• Vision-language alignment (VLA) enables zero-shot learning but existing methods show a gap between visual and textual embeddings.
• Bridging this gap is crucial for improving automated diagnostic capabilities in medical imaging.

Purpose of the Study:

• To propose a novel framework, Bridged Semantic Alignment (BrgSA), to bridge the gap between visual and textual embeddings in medical imaging.
• To enhance zero-shot learning capabilities for medical image diagnosis, particularly for underrepresented abnormalities.
• To improve the alignment and interaction between visual and language modalities in clinical practice.

Main Methods:

• Utilized a large language model for semantic summarization of clinical reports to extract high-level semantic information.
• Designed a Cross-Modal Knowledge Interaction module with a knowledge bank to act as a semantic bridge between modalities.
• Constructed a benchmark dataset including 15 underrepresented abnormalities and utilized two existing datasets for comprehensive evaluation.

Main Results:

• BrgSA achieved state-of-the-art performance on both public and custom benchmark datasets.
• Demonstrated significant improvements in the zero-shot diagnosis of underrepresented abnormalities.
• Successfully narrowed the gap between visual and textural embeddings, enhancing cross-modal alignment.

Conclusions:

• The proposed BrgSA framework effectively bridges the semantic gap in vision-language alignment for medical imaging.
• BrgSA offers a powerful solution for zero-shot medical image diagnosis, especially for rare conditions.
• This approach holds significant potential for advancing automated diagnosis in clinical practice by leveraging diverse data and reducing annotation burden.