LLM-Mediated Data Extraction from Patient Records after Radical Prostatectomy

AbstractIntegrating large language models (LLMs) into health care has the potential to improve data extraction from electronic health records (EHRs). In this study, we assess the extraction accuracy of our LLM-powered tool, the National Institutes of Health (NIH) Integrated Data Analysis Platform Text Extraction Program (NTEP), on radical prostatectomy pathology reports. A total of 369 patients enrolled in a clinical trial (NCT02594202) who underwent radical prostatectomy at the NIH were included. We compared a manually curated ground-truth dataset containing 4797 data points for 13 variables abstracted from radical prostatectomy pathology reports to a dataset created using NTEP, a platform that allows researchers to develop prompts aimed at extracting data directly from unstructured EHR documents using the Generative Pretrained Transformer 4 LLM. Comparison between extraction methods was assessed using Cohen's kappa for binary and categorical variables and mean squared error (MSE) with Spearman's correlation for continuous and discrete variables. Extraction accuracy was calculated for each variable, and 95% confidence intervals were adjusted using the Agresti-Coull method to account for sampling variability. Overall, 4788 of 4797 (99.8%) data points were correctly extracted by the LLM. With discrete variables, the LLM demonstrated a 100% match between datasets for the number of positive lymph nodes but miscalculated the total number of lymph nodes, resulting in an MSE of 0.0027 and a Spearman's coefficient of 0.99. For continuous variables, the LLM demonstrated a 100% match with the ground-truth dataset for prostate dimensions but failed to identify prostate weight in one patient. NTEP demonstrated 99.8% accuracy in extracting data from radical prostatectomy pathology reports, underscoring its effectiveness in accurately and securely processing variables from unstructured EHR data. Through robust prompt engineering, NTEP reliably extracted both discrete and continuous variables, with only minor errors identified. These results highlight its potential for enhancing automated EHR data extraction while indicating areas for further refinement.