HEART: Learning better representation of EHR data with a heterogeneous relation-aware transformer

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Machine Learning

Background:

• Electronic Health Records (EHRs) contain valuable patient data but existing language models struggle to capture complex relationships between diverse medical entities.
• Current methods often treat medical entities (diagnoses, medications, procedures, lab tests) homogeneously or focus only on diagnoses, limiting performance.
• There is a need for advanced EHR representation learning that explicitly models the heterogeneous correlations among different medical entity types.

Purpose of the Study:

• To develop a foundational language model pre-trained on EHR data that explicitly incorporates heterogeneous correlations among medical entities.
• To improve EHR representation learning by moving beyond homogeneous encoding and single-entity focus.

Main Methods:

• Propose HEART (Heterogeneous Relation-aware Transformer for EHR), a novel transformer model designed for EHR data.
• Incorporate heterogeneous entities and represent pairwise relationships using relation embeddings for complex reasoning.
• Employ a multi-level attention scheme to efficiently connect encounters and utilize two pretraining tasks: missing entity prediction and anomaly detection.

Main Results:

• HEART demonstrates superior performance over four state-of-the-art foundation models on two EHR datasets and five downstream tasks.
• Achieved significant improvements in death (12.1%) and readmission (4.1%) prediction compared to Med-BERT.
• Case studies confirm HEART's ability to provide interpretable insights into entity relationships via learned embeddings.

Conclusions:

• HEART effectively leverages heterogeneous relationships between medical entities in EHRs for improved representation learning.
• The model's multi-level encoding and specialized pretraining objectives enhance efficiency and effectiveness.
• Experimental results validate HEART's strong performance and practical utility in healthcare applications.