Generative Artificial Intelligence Successfully Automates Data Extraction From Unstructured Magnetic Resonance...

Radiology reports are stored as plain text in most electronic health records, rendering the data computationally inaccessible. Large language models are powerful tools for analyzing unstructured text but relatively untested in urologic oncology. We aimed to develop a pipeline to extract data from plain text prostate magnetic resonance imaging (MRI) reports using GPT4.0 and compare the accuracy to manually abstracted data.

We developed a data pipeline using a secure, enterprise-wide deployment of OpenAI's GPT-4.0 to automatically extract data elements from prostate MRI report text when presented with prostate MRI reports. Identical prompts and reports were sent multiple times to determine response variability. We extracted 15 data elements per report and compared accuracy to a manually abstracted gold standard.

Across 424 prostate MRI reports, GPT-4.0 response accuracy was consistently above 95%. Individual field accuracies were 98.3% (96.3%-99.3%) for prostate-specific antigen density, 97.4% (95.4%-98.7%) for extracapsular extension, and 98.1% (96.3%-99.2%) for TNM stage, and had a median of 98.1% (96.3%-99.2%), a mean of 97.2% (95.2%-98.3%), and a range of 99.8% (98.7%-100.0%) for number of suspicious lesions to 87.7% (84.2%-90.7%) for identification of lesion location in the base of the prostate. Response variability over five repeated runs ranged from 0.14% to 3.61%, differed based on the data element extracted (P < .001), and was inversely correlated with accuracy (P < .001). In disagreements between manual and GPT-4.0 extracted data, GPT-4.0 responses were more often deemed correct by an additional reviewer.

GPT-4.0 had high accuracy with low variability in extracting data points from prostate cancer MRI reports with low upfront programming requirements. This represents an effective tool to expedite medical data extraction for clinical and research use cases.