A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Biomedical Data Science

Background:

• Deep learning models for clinical decision support often rely solely on structured electronic health records (EHR).
• Narrative clinical notes contain valuable complementary information but are frequently underutilized in predictive models.
• Accurate prediction of in-hospital mortality is crucial for timely medical intervention and resource allocation.

Purpose of the Study:

• To develop a novel multimodal transformer model for improved prediction of in-hospital mortality.
• To integrate both structured EHR data and unstructured clinical notes into a unified predictive framework.
• To enhance model interpretability through feature visualization techniques.

Main Methods:

• A multimodal transformer architecture was designed to fuse information from clinical notes and structured EHR data.
• Integrated Gradients (IG) and Shapley values were employed for feature selection and interpretability, identifying critical words and EHR features.
• Clinical BERT was utilized for learning representations of clinical notes, with investigations into domain adaptive pretraining and task adaptive fine-tuning.

Main Results:

• The proposed multimodal model demonstrated superior performance in predicting in-hospital mortality compared to existing methods.
• Achieved an Area Under the Precision-Recall Curve (AUCPR) of 0.538, an Area Under the Receiver Operating Characteristic Curve (AUCROC) of 0.877, and an F1-score of 0.490.
• Visualization of important words and clinical features provided insights into the model's decision-making process.

Conclusions:

• Integrating narrative clinical notes with structured EHR data significantly enhances the prediction of in-hospital mortality.
• The developed multimodal transformer model offers a promising approach for advanced clinical decision support.
• The interpretability methods provide valuable tools for understanding and trusting AI-driven predictions in healthcare settings.