A multimodal generative model for structured and unstructured electronic health records.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Biomedical Data Science

Background:

• Electronic Health Records (EHRs) contain heterogeneous data (structured and unstructured) crucial for clinical insights.
• Current AI models struggle to effectively utilize this multimodal EHR data, often losing critical temporal or quantitative details.
• Existing methods typically focus on either structured data or text serialization, limiting generative capabilities.

Purpose of the Study:

• To develop a multimodal generative model, Generative Deep Patient (GDP), capable of jointly processing structured EHR time-series and unstructured clinical texts.
• To leverage advanced deep learning architectures, including CNN-Transformer encoders and LLaMA-based decoders, for unified EHR data modeling.
• To evaluate GDP's performance on downstream tasks including clinical prediction and narrative generation using the MIMIC-IV dataset.

Main Methods:

• GDP employs a CNN-Transformer encoder for structured EHR data and fuses it with text representations via cross-modal attention.
• A Large Language Model Meta AI (LLaMA)-based generative decoder is utilized for multimodal data synthesis.
• The model is trained with generative pretraining, auxiliary temporal objectives, and multi-task fine-tuning for prediction and narrative generation.

Main Results:

• GDP achieved high AUROC scores for predicting heart failure (0.923) and type 2 diabetes (0.817), and 30-day readmission (0.627).
• For narrative generation, GDP produced clinically coherent discharge summaries, achieving ROUGE-L of 0.135 and BERTScore-F1 of 0.545.
• Human evaluations confirmed GDP's high faithfulness, fluency, and clinical utility in generated content.

Conclusions:

• Unified multimodal generative modeling of structured EHR data and clinical text is feasible and effective.
• GDP demonstrates competitive performance across diverse downstream tasks, including prediction and text generation.
• This work paves the way for advanced EHR-scale multimodal AI models in clinical practice.