Safety and diagnostic accuracy of large-language model application of PECARN head injury algorithm

Clinical decision algorithms used by clinicians guide evidence-based decisions and actions. Automated tools can help with the adoption and sustainability of these algorithms, and automation is especially needed in the emergency setting which requires quick decision making in a chaotic environment. This preimplementation study evaluated whether a large language model (LLM) can apply the Pediatric Emergency Care Applied Research Network (PECARN) Head Injury Algorithm safely and accurately on emergency department notes, a preliminary step in the development of a clinical decision support tool for treating pediatric head injuries.

We studied the safety and capability of an LLM, Generative Pretrained Transformer (GPT), to apply the PECARN Head Injury Algorithm using notes of patient encounters aged 3 months - 18 years who presented to a pediatric emergency department (ED) with chief complaint of "head injury". A dataset of 24 patients was curated to include a diverse range of symptoms for developing models and a dataset of 122 was randomly selected for testing. We developed and compared four LLM models to extract clinical features from clinical notes. Primary outcomes were safety, measured as negative predictive value (NPV), and accuracy of the LLM models compared to gold standard pediatric emergency medicine (PEM) physicians. Secondary outcomes were the accuracies for the nine features used in the PECARN algorithm.

All models demonstrated high NPV comparable to PEM physicians. The GPT model with the highest combination of NPV and accuracy was the prompt-engineered "Optimized Features Model" (NPV = 0.98, accuracy = 0.89), which performed similarly to that of the ED clinicians in both NPV (0.99) and accuracy (0.92).

Our LLM-based tool for a clinical decision algorithm demonstrated high accuracy and NPV. While promising, further studies on scalability and feasibility are needed to ensure LLM-based digital health tools encourage safe, effective care for pediatric patients in the ED.