Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Electronic health records (EHR) contain both structured (clinical codes) and unstructured (clinical narratives) data.
• Existing methods often analyze these data modalities separately, overlooking their inherent interactions.
• Understanding these interactions is crucial for comprehensive patient data analysis.

Purpose of the Study:

• To develop a Medical Multimodal Pre-trained Language Model (MedM-PLM) that effectively learns from and models interactions between structured and unstructured EHR data.
• To enhance EHR representations by capturing the interplay between clinical codes and narratives.
• To improve downstream clinical tasks through multimodal data integration.

Main Methods:

• Proposed MedM-PLM, a Transformer-based neural network architecture.
• Utilized two Transformer components to process individual data modalities.
• Incorporated a cross-modal module to explicitly model interactions between structured and unstructured data.
• Pre-trained the model on the MIMIC-III dataset.

Main Results:

• MedM-PLM demonstrated superior performance on three downstream clinical tasks: medication recommendation, 30-day readmission prediction, and ICD coding.
• The model significantly outperformed state-of-the-art methods.
• Further analyses confirmed the model's robustness and interpretability.

Conclusions:

• MedM-PLM effectively captures the intrinsic interactions between structured and unstructured EHR data.
• The proposed model offers enhanced EHR representations, leading to improved performance in clinical decision-making tasks.
• This approach has the potential to provide more comprehensive interpretations for clinical decision-making.