Robust multi-modal fusion architecture for medical data with knowledge distillation.

Area of Science:

• Artificial Intelligence in Medicine
• Deep Learning for Healthcare
• Multi-modal Data Fusion

Background:

• Multi-modal data fusion significantly improves deep learning model performance in medical applications.
• Missing data modalities are a common challenge in medical datasets, hindering model generalization.

Purpose of the Study:

• To develop an efficient multi-modal fusion framework for medical data robust to missing modalities.
• To maintain consistent performance in deep learning models despite data incompleteness.

Main Methods:

• Fused chest X-rays, clinical notes, and tabular data using a pooled bottleneck (PB) attention module.
• Employed knowledge distillation (KD) and gradient modulation (GM) to enhance inference and training stability.
• Evaluated on MIMIC-IV dataset for in-hospital mortality prediction, using AUROC and AUPRC metrics.

Main Results:

• Achieved AUROC of 0.886 and AUPRC of 0.459, outperforming baseline models.
• Demonstrated consistent performance even with one or two missing modalities.
• Ablation studies confirmed the significant contribution of PB, KD, and GM components.

Conclusions:

• The proposed framework offers a robust solution for multi-modal fusion in medical data, effectively handling missing modalities.
• This approach shows promise for improving patient outcome prediction and can be extended to more data types.