Natural Language Processing for Risk Stratification of Emergency Department Patients with Chest Pain

Natural language processing can transform unstructured clinical data into actionable insights that optimize workflows, improve patient outcomes, and reduce bias. We created a machine learning model that predicted suspicion of acute coronary syndrome (ACS) based on physician notes and assessed its capability to predict major adverse cardiovascular events (MACE) when implemented within the HEART (history, electrocardiogram, age, risk factors, troponin) score.

We retrospectively reviewed adults presenting with chest pain to The Ottawa Hospital emergency departments (EDs). GPT-4 produced and extracted 11 binary symptom features from physician-documented history of present illness. An extreme gradient boosting (XGBoost) classifier was trained on these features to predict ED disposition, a surrogate marker of physician suspicion of ACS (pACS). The resulting natural language processing-derived ACS suspicion score (nACS) replaced the history component of HEART (nHEART) and was compared with the disposition-based composite (pHEART).

Of 1011 patients (48.5% female; mean age, 55.7 years), 69 (6.8%) experienced MACE within 6 weeks. nHEART achieved sensitivity of 97.1% and specificity of 53.8%; pHEART achieved sensitivity and specificity of 95.7% and 69.1%, respectively. pACS and troponin were the strongest multivariable predictors. GPT-4o extraction showed Fleiss' kappa > 0.8 across runs and Cohen's kappa > 0.6 vs extraction by research assistants.

Large language models can extract structured symptom features from ED notes, and an XGBoost model trained on these features can reproduce a coarse measure of physician suspicion of ACS. Clinical deployment will require further model training and testing against a prospectively documented measure of physician ACS suspicion.