Automated Detection and Classification of Radiology Report Discrepancies Using NLP: A Tool for Resident Education and...

The aim of this study was to develop and evaluate a natural language processing (NLP) system that automatically detects and classifies discrepancies between preliminary and final radiology reports, with the goal of enhancing resident education through structured feedback.

A total of 889 deidentified lumbar spine MRI reports (768 with revisions) from December 2023 to March 2024 were retrospectively analyzed. Preliminary full diagnostic reports were generated by trainee residents during daytime rotations; final reports were subsequently verified by attending radiologists remotely. Discrepancies in the diagnostic impression section were extracted using a multistep NLP pipeline: sentence segmentation, Bidirectional Encoder Representations From Transformers-based sentence matching, GPT-4-based named entity recognition, and rule-based classification into 11 correction types (missed diagnosis, misdiagnosis, missed image feature, misidentified image feature, localization error, diagnostic reasoning error, clinical query omission, severity error, confidence difference, typographic error, and terminology refinement). Ground truth was established by three radiologists. System performance was evaluated for each correction type individually using accuracy, sensitivity, specificity, and intraclass coefficient. Trends in resident and attending radiologist performance were analyzed at the report level.

The NLP system achieved high accuracy (0.983-0.999), sensitivity (0.977-1.000), and specificity (0.900-1.000) for each of the 11 correction types, with strong interrater reliability (intraclass correlation coefficient > 0.75). Most common corrections were misdiagnosis (504 of 768 [65.6%]) and missed diagnosis (356 of 768 [46.4%]). Residents showed significant variability in error rates, especially in missed diagnosis (range, 11.1%-59.1% across 16 residents) and misdiagnosis (range, 24.0%-71.1% across 16 residents). Attending radiologists exhibited marked heterogeneity in correction patterns (n = 6; individual workload range, 95-187 reports; median, 159 reports), with significant variability across all major error types (P < .001 for missed diagnosis [20.6%-82.0%], misdiagnosis [31.4%-66.7%], localization error [15.8%-54.7%], and terminology refinement [3.2%-36.7%]).

The NLP-based discrepancy tracking system accurately identifies and classifies report modifications, enabling scalable, targeted feedback for radiology residents. Variability among residents and attending radiologists highlights the need for individualized training and standardized review practices.