Trajectory-Ordered Objectives for Self-Supervised Representation Learning of Temporal Healthcare Data Using...

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Machine Learning

Background:

• Electronic health records (EHRs) offer vast potential for improving patient care through deep learning.
• Modeling sequential EHR data is challenging due to complex temporal relationships in patient trajectories.
• Existing transformer models with masked language modeling (MLM) capture context but struggle with temporal dynamics.

Purpose of the Study:

• To enhance EHR sequence modeling by addressing limitations in capturing temporal dependencies.
• To introduce a novel transformer-based model, TOO-BERT, for improved understanding of patient trajectories.

Main Methods:

• Developed Trajectory Order Objective BERT (TOO-BERT), a transformer model integrating a novel Trajectory Order Objective (TOO) with MLM pretraining.
• TOO-BERT pretrains by distinguishing ordered from permuted medical event sequences, focusing on frequently co-occurring codes/visits.
• Evaluated TOO-BERT on MIMIC-IV and Malmo Diet and Cancer Cohort (MDC) datasets, comparing against conventional methods and MLM-pretrained transformers.

Main Results:

• TOO-BERT significantly outperformed existing methods in predicting heart failure (HF), Alzheimer's disease (AD), and prolonged length of stay (PLS) on both datasets.
• Achieved improved AUC scores for HF and AD prediction on the MDC dataset (e.g., HF from 67.7% to 73.9%).
• Demonstrated robust performance in HF prediction even with limited fine-tuning data on the MIMIC-IV dataset.

Conclusions:

• Integrating temporal ordering objectives into MLM-pretrained models effectively captures complex temporal relationships in EHR data.
• TOO-BERT provides deeper insights into disease progression by representing sophisticated structural patterns in patient trajectories.
• The model offers a more nuanced understanding of patient health journeys and disease development.