Related Experiment Video
Updated: Jan 8, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Development and validation of a pre-trained language model for neonatal morbidities: a retrospective, multicentre,
Feng Xie1, Philip Chung2, Jonathan D Reiss3
1Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA, USA; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University, Stanford, CA, USA; Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA.
Insights
NeonatalBERT, a large language model (LLM), effectively estimates neonatal morbidity risk from clinical notes. This AI tool shows promise for improving newborn care and hospital efficiency.
Area of Science:
- Artificial Intelligence in Medicine
- Clinical Informatics
- Neonatal Health
Background:
- Early identification of neonatal morbidities is crucial for timely interventions.
- Traditional methods using biomarkers and tabular data have limitations in assessing newborn morbidity risk.
- Large Language Models (LLMs) offer potential for analyzing unstructured clinical notes.
Purpose of the Study:
- To develop and evaluate NeonatalBERT, a novel LLM for estimating neonatal morbidity risk.
- To compare NeonatalBERT's performance against existing LLMs and traditional models.
- To assess the utility of clinical notes for predicting neonatal complications.
Main Methods:
- A prognostic study using retrospective primary and external cohorts from two US academic medical centers.
- NeonatalBERT was pre-trained and fine-tuned on clinical notes.
- Performance was evaluated against BioBERT, Bio-ClinicalBERT, and traditional models using Area Under the Precision-Recall Curve (AUPRC) and F1 scores for 19 neonatal morbidities.
Main Results:
- NeonatalBERT achieved a mean AUPRC of 0.291 in the primary cohort and 0.360 in the external cohort, outperforming other models.
- In the primary cohort, NeonatalBERT outperformed Bio-ClinicalBERT (0.238), BioBERT (0.217), and traditional models (0.194).
- In the external cohort, NeonatalBERT's mean AUPRC (0.360) surpassed the range of other models (0.224-0.333).
Conclusions:
- NeonatalBERT effectively estimates neonatal morbidity risk from unstructured clinical notes.
- The model demonstrates significant potential for enhancing neonatal care.
- This AI approach can help streamline hospital operations and improve patient outcomes.
Background:
Early identification and monitoring of neonatal morbidities are critical for timely interventions that can prevent complications, optimise resource use, and support families. Although traditional tools based on tabular data and biomarkers are beneficial, they are restricted in assessing the risk of morbidities in newborns. In this study, we developed NeonatalBERT, a pre-trained large language model (LLM) that estimates the risk of neonatal morbidities from clinical notes.
Methods:
This prognostic study investigated retrospective primary and external cohorts from two different quaternary-care academic medical centres in the USA: Stanford Health Care and Beth Israel Deaconess Medical Center. NeonatalBERT was initially pre-trained on clinical notes from the primary cohort and then fine-tuned separately for both cohorts. NeonatalBERT was also compared against other existing LLMs, such as BioBERT and Bio-ClinicalBERT, as well as traditional machine learning and logistic regression models using tabular features. NeonatalBERT was evaluated on 19 neonatal morbidities (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, pulmonary hypertension, atelectasis, aspiration syndrome, intraventricular haemorrhage, periventricular leukomalacia, neonatal seizures, other CNS disorders, patent ductus arteriosus, cardiovascular instability, sepsis, candidiasis, anaemia, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death) for the primary cohort and ten for the external cohort (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, intraventricular haemorrhage, patent ductus arteriosus, sepsis, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death). For each outcome, the area under the receiver operating characteristic curve, area under the precision-recall curve (AUPRC), and F1 scores were evaluated.
Findings:
32 321 newborns were included in the primary cohort, including 27 411 in the primary training set (mean gestational age 38·64 weeks [SD 2·30]; 13 056 [47·6%] female and 14 355 [52·4%] male newborns) and 4910 in the primary testing set (mean gestational age 38·64 [2·13] weeks; 2336 [47·6%] female and 2574 [52·4%] male newborns). Additionally, 7061 newborns were selected into the external cohort, including 5653 in the external training set (1567 [27·7%] premature and 4086 [72·3%] term births; 2614 [46·2%] female and 3039 [53·8%] male newborns) and 1408 in the external testing set (383 [27·2%] premature and 1025 [72·8%] term births; 624 [44·3%] female and 784 [55·7%] male newborns). In the primary cohort, the mean AUPRC over 19 outcomes was 0·291 (95% CI 0·268-0·314) for NeonatalBERT, 0·238 (0·217-0·259) for Bio-ClinicalBERT, 0·217 (0·197-0·236) for BioBERT, and 0·194 (0·177-0·211) for the traditional model using tabular data. In the external cohort, NeonatalBERT had a mean AUPRC of 0·360 (0·328-0·393), outperforming other models with the range of 0·224-0·333.
Interpretation:
Based on validation using two large-scale US datasets, NeonatalBERT effectively estimates the risk of neonatal morbidities from unstructured clinical notes of newborns. The promising results from this study show the potential of NeonatalBERT to enhance neonatal care and streamline hospital operations.
Funding:
National Institutes of Health, Burroughs Wellcome Fund, March of Dimes Foundation, Alfred E Mann Foundation, Gates Foundation, Christopher Hess Research Fund, Roberts Foundation Research Fund, Prematurity Research Center, and Stanford Maternal & Child Health Research Institute Postdoctoral Support funds.
