Missing-modality enabled multi-modal fusion architecture for medical data.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Deep Learning

Background:

• Multi-modal data fusion enhances deep learning model performance.
• Missing data in medical applications hinders multi-modal model effectiveness.
• Adapting models to handle missing modalities is crucial for clinical utility.

Purpose of the Study:

• Develop a robust multi-modal fusion architecture for medical data.
• Improve model performance on clinical tasks despite missing modalities.
• Enhance robustness to missing data during inference.

Main Methods:

• Fused X-ray, radiology reports, and tabular data using Transformer-based bi-modal fusion modules.
• Combined three bi-modal modules into a tri-modal fusion framework.
• Employed multivariate loss functions and conducted comparative/ablation experiments on MIMIC-IV and MIMIC-CXR datasets.

Main Results:

• Achieved superior predictive performance with average AUROC/AUPRC of 0.916/0.551 (14-label task) and 0.816/0.392 (mortality prediction).
• Demonstrated slight performance decrease with incomplete data, highlighting robustness.
• Validated effectiveness and component contributions through rigorous experiments.

Conclusions:

• The proposed architecture effectively fuses multiple medical data modalities.
• Exhibited strong robustness to missing modalities, crucial for real-world clinical applications.
• Shows potential for scaling to more modalities, increasing clinical practicality.