Core-Periphery Multi-Modality Feature Alignment for Zero-Shot Medical Image Analysis.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Multi-modality learning, using models like CLIP, excels at zero-shot tasks.
• Direct application of CLIP to medical imaging suffers from domain shift, degrading performance.
• Disparities between natural and medical images hinder CLIP's effectiveness in healthcare.

Purpose of the Study:

• To enhance CLIP's zero-shot capabilities for medical image analysis.
• To develop a novel approach for joint modeling of medical images and clinical text.
• To improve the accuracy and interpretability of AI in medical diagnostics.

Main Methods:

• Introduced Core-Periphery feature alignment for CLIP (CP-CLIP).
• Designed an auxiliary neural network based on the core-periphery principle.
• Aligned medical image and text features into a unified latent space using brain network organization principles.

Main Results:

• CP-CLIP effectively mitigates domain shift issues in medical image analysis.
• Achieved significant improvements in CLIP's zero-shot performance for medical tasks.
• Demonstrated excellent explanatory capability, identifying critical disease-related regions.
• Outperformed existing methods across five diverse public medical datasets.

Conclusions:

• CP-CLIP offers a superior approach for multimodal feature alignment in medical AI.
• The method enhances zero-shot medical image prediction and critical feature detection.
• CP-CLIP shows significant promise for advancing AI applications in clinical analysis.