Benchmarking of pre-training strategies for electronic health record foundation models

Our objective is to compare different pre-training strategies for electronic health record (EHR) foundation models.

We evaluated three approaches using a transformer-based architecture: baseline (no pre-training), self-supervised pre-training with masked language modeling, and supervised pre-training. The models were assessed on their ability to predict both major adverse cardiac events and mortality occurring within 12 months. The pre-training cohort was 405 679 patients prescribed antihypertensives and the fine tuning cohort was 5525 patients who received doxorubicin.

Task-specific supervised pre-training achieved superior performance (AUROC 0.70, AUPRC 0.23), outperforming both self-supervised pre-training and the baseline. However, when the model was evaluated on the task of 12-month mortality prediction, the self-supervised model performed best.

While supervised pre-training excels when aligned with downstream tasks, self-supervised approaches offer more generalized utility.

Pre-training strategy selection should consider intended applications, data availability, and transferability requirements.