Explainable multimodal deep learning models for variable-length sequences in critically ill patients | JoVE Visualize

Area of Science:

Artificial Intelligence in Medicine
Clinical Informatics
Deep Learning for Healthcare

Background:

Deep learning models excel at predicting clinical events using structured electronic health record (EHR) data.
Incorporating unstructured clinical notes enhances model accuracy but presents challenges in multimodal fusion and explainability, especially with variable-length temporal data.
Existing methods struggle to effectively integrate diverse data types and provide interpretable predictions for intensive care unit (ICU) trajectories.

Purpose of the Study:

To develop an explainable temporal modeling framework for multimodal EHR data.
To accommodate variable-length ICU trajectories and support diverse outcome prediction tasks.
To improve the accuracy and interpretability of deep learning models in critical care.

Main Methods:

Two multimodal recurrent neural networks (RNNs) with distinct fusion architectures (Pre-RNN and Post-RNN) were developed, integrating structured EHR variables and unstructured clinical notes at hourly timesteps.
Time2Vec and RNN layers with masking handled variable-length sequences, while integrated gradients were used for explainability, quantifying temporal and cross-modal feature importance.
Models were evaluated on predicting 24-hour mortality, seven-day discharge, and four-hour ventilator or vasopressor onset using a public EHR dataset.

Main Results:

Multimodal fusion models significantly outperformed unimodal baselines across all prediction tasks.
The Pre-RNN fusion architecture achieved the highest area under the precision-recall curve (AUPRC) for three of the four outcomes.
Performance gains were more substantial for intermediate and long-horizon events (≥24 hours) compared to short-horizon events.
Integrated gradients analysis revealed specific attribution patterns, linking physiological features and clinical concepts to patient risk.

Conclusions:

The developed variable-length multimodal framework enhances the performance of deep learning models in critical care.
The framework provides timestep-level feature importance, significantly improving the explainability and clinical relevance of predictions.
This approach offers a promising direction for advancing AI applications in critical care settings.