DeAF: A multimodal deep learning framework for disease prediction.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Machine Learning

Background:

• Multimodal deep learning models face training challenges due to conflicts between sub-models and fusion modules.
• Effective integration of diverse data modalities (e.g., medical images, clinical data) is crucial for accurate disease prediction.

Purpose of the Study:

• To propose a novel framework, DeAF (Decoupling Feature Alignment and Fusion), to address training difficulties in multimodal deep learning.
• To enhance the accuracy of disease prediction by improving the interaction between medical image features and clinical data.

Main Methods:

• The DeAF framework employs a two-stage training process: unsupervised representation learning with a modality adaptation (MA) module for feature alignment, followed by supervised learning with a self-attention fusion (SAF) module.
• The framework was applied to predict postoperative efficacy in colorectal cancer patients and the progression of Mild Cognitive Impairment (MCI) to Alzheimer's disease.

Main Results:

• The DeAF framework demonstrated significant improvements in predictive performance compared to existing methods.
• Ablation experiments validated the effectiveness and rationale of the proposed framework's components.

Conclusions:

• The DeAF framework effectively enhances the integration of local medical image features and clinical data.
• This leads to the derivation of more discriminative multimodal features, significantly improving disease prediction accuracy.