Dual-contrastive modality recovery for incomplete multi-modal brain disease diagnosis

Area of Science:

• Neuroscience
• Artificial Intelligence
• Medical Imaging

Background:

• Multi-modal learning aids brain disease diagnosis (epilepsy, Alzheimer's).
• Incomplete data and lack of cross-modal alignment hinder current methods.
• Existing approaches often focus on data reconstruction, neglecting semantic consistency.

Purpose of the Study:

• To introduce BrainCLIP, a two-stage framework for incomplete multi-modal learning in brain disease diagnosis.
• To address challenges of missing data and enhance cross-modal semantic alignment.
• To improve diagnosis of epilepsy and Alzheimer's disease.

Main Methods:

• Proposed BrainCLIP, a CLIP-inspired two-stage framework.
• Implemented a multi-modal contrastive learning stage for aligning text, fMRI, and DTI.
• Employed dual contrastive recovery (modality and sample level) for feature refinement.
• Fused recovered and available modalities with textual embeddings for classification.

Main Results:

• BrainCLIP demonstrated effectiveness in diagnosing epilepsy and Alzheimer's disease.
• The dual contrastive recovery strategy ensured semantically consistent and discriminative recovered features.
• Joint design of recovery and representation learning improved performance with incomplete data.

Conclusions:

• BrainCLIP offers a robust solution for brain disease diagnosis with incomplete multi-modal data.
• The framework successfully achieves cross-modal semantic alignment and accurate data recovery.
• This approach advances the application of AI in clinical neuroscience.