Explainable multimodal deep learning models for variable-length sequences in critically ill patients

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Deep Learning for Healthcare

Background:

• Deep learning models excel at predicting clinical events using structured electronic health record (EHR) data.
• Incorporating unstructured clinical notes enhances model accuracy but presents challenges in multimodal fusion and explainability, especially with variable-length temporal data.
• Existing methods struggle to effectively integrate diverse data types and provide interpretable predictions for intensive care unit (ICU) trajectories.

Purpose of the Study:

• To develop an explainable temporal modeling framework for multimodal EHR data.
• To accommodate variable-length ICU trajectories and support diverse outcome prediction tasks.
• To improve the accuracy and interpretability of deep learning models in critical care.

Main Methods:

• Two multimodal recurrent neural networks (RNNs) with distinct fusion architectures (Pre-RNN and Post-RNN) were developed, integrating structured EHR variables and unstructured clinical notes at hourly timesteps.
• Time2Vec and RNN layers with masking handled variable-length sequences, while integrated gradients were used for explainability, quantifying temporal and cross-modal feature importance.
• Models were evaluated on predicting 24-hour mortality, seven-day discharge, and four-hour ventilator or vasopressor onset using a public EHR dataset.

Main Results:

• Multimodal fusion models significantly outperformed unimodal baselines across all prediction tasks.
• The Pre-RNN fusion architecture achieved the highest area under the precision-recall curve (AUPRC) for three of the four outcomes.
• Performance gains were more substantial for intermediate and long-horizon events (≥24 hours) compared to short-horizon events.
• Integrated gradients analysis revealed specific attribution patterns, linking physiological features and clinical concepts to patient risk.

Conclusions:

• The developed variable-length multimodal framework enhances the performance of deep learning models in critical care.
• The framework provides timestep-level feature importance, significantly improving the explainability and clinical relevance of predictions.
• This approach offers a promising direction for advancing AI applications in critical care settings.