Related Experiment Video
Updated: Jan 8, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Development and validation of a pre-trained language model for neonatal morbidities: a retrospective, multicentre,
Feng Xie1, Philip Chung2, Jonathan D Reiss3
1Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA, USA; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University, Stanford, CA, USA; Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA.
Background:
Early identification and monitoring of neonatal morbidities are critical for timely interventions that can prevent complications, optimise resource use, and support families. Although traditional tools based on tabular data and biomarkers are beneficial, they are restricted in assessing the risk of morbidities in newborns. In this study, we developed NeonatalBERT, a pre-trained large language model (LLM) that estimates the risk of neonatal morbidities from clinical notes.
Methods:
This prognostic study investigated retrospective primary and external cohorts from two different quaternary-care academic medical centres in the USA: Stanford Health Care and Beth Israel Deaconess Medical Center. NeonatalBERT was initially pre-trained on clinical notes from the primary cohort and then fine-tuned separately for both cohorts. NeonatalBERT was also compared against other existing LLMs, such as BioBERT and Bio-ClinicalBERT, as well as traditional machine learning and logistic regression models using tabular features. NeonatalBERT was evaluated on 19 neonatal morbidities (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, pulmonary hypertension, atelectasis, aspiration syndrome, intraventricular haemorrhage, periventricular leukomalacia, neonatal seizures, other CNS disorders, patent ductus arteriosus, cardiovascular instability, sepsis, candidiasis, anaemia, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death) for the primary cohort and ten for the external cohort (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, intraventricular haemorrhage, patent ductus arteriosus, sepsis, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death). For each outcome, the area under the receiver operating characteristic curve, area under the precision-recall curve (AUPRC), and F1 scores were evaluated.
Findings:
32 321 newborns were included in the primary cohort, including 27 411 in the primary training set (mean gestational age 38·64 weeks [SD 2·30]; 13 056 [47·6%] female and 14 355 [52·4%] male newborns) and 4910 in the primary testing set (mean gestational age 38·64 [2·13] weeks; 2336 [47·6%] female and 2574 [52·4%] male newborns). Additionally, 7061 newborns were selected into the external cohort, including 5653 in the external training set (1567 [27·7%] premature and 4086 [72·3%] term births; 2614 [46·2%] female and 3039 [53·8%] male newborns) and 1408 in the external testing set (383 [27·2%] premature and 1025 [72·8%] term births; 624 [44·3%] female and 784 [55·7%] male newborns). In the primary cohort, the mean AUPRC over 19 outcomes was 0·291 (95% CI 0·268-0·314) for NeonatalBERT, 0·238 (0·217-0·259) for Bio-ClinicalBERT, 0·217 (0·197-0·236) for BioBERT, and 0·194 (0·177-0·211) for the traditional model using tabular data. In the external cohort, NeonatalBERT had a mean AUPRC of 0·360 (0·328-0·393), outperforming other models with the range of 0·224-0·333.
Interpretation:
Based on validation using two large-scale US datasets, NeonatalBERT effectively estimates the risk of neonatal morbidities from unstructured clinical notes of newborns. The promising results from this study show the potential of NeonatalBERT to enhance neonatal care and streamline hospital operations.
Funding:
National Institutes of Health, Burroughs Wellcome Fund, March of Dimes Foundation, Alfred E Mann Foundation, Gates Foundation, Christopher Hess Research Fund, Roberts Foundation Research Fund, Prematurity Research Center, and Stanford Maternal & Child Health Research Institute Postdoctoral Support funds.
