Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers

Area of Science:

• Artificial Intelligence in Medicine
• Multimodal Deep Learning
• Clinical Decision Support Systems

Background:

• Current machine learning diagnostic tools often rely on single data types, limiting comprehensive patient assessment.
• Integrating diverse patient data, including imaging and nonimaging information, is crucial for accurate clinical diagnosis.

Purpose of the Study:

• To develop and evaluate a novel transformer-based neural network architecture for multimodal data integration.
• To compare the diagnostic performance of this multimodal model against single-modality models for up to 25 conditions.

Main Methods:

• Retrospective analysis of chest radiographs and clinical data from MIMIC and internal ICU databases (2008-2020).
• Training a transformer-based neural network on nonimaging data, imaging data, or both.
• Performance assessment using area under the receiver operating characteristic curve (AUC).

Main Results:

• The multimodal model demonstrated superior diagnostic performance across all evaluated pathologic conditions.
• For the MIMIC dataset, mean AUC was 0.77 (multimodal) vs. 0.70 (imaging only) and 0.72 (nonimaging only).
• Similar improvements were observed in the internal dataset, confirming the multimodal model's efficacy.

Conclusions:

• A neural network integrating both imaging and nonimaging data significantly enhances disease diagnosis accuracy in ICU patients.
• Multimodal data fusion offers a more robust approach for diagnosing multiple conditions compared to single-modality models.