Automated Detection and Classification of Radiology Report Discrepancies Using NLP: A Tool for Resident Education and...

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Radiology resident education relies on feedback from preliminary reports.
• Identifying and classifying discrepancies between preliminary and final reports is crucial for learning.
• Current methods for discrepancy analysis are often manual and time-consuming.

Purpose of the Study:

• To develop and evaluate a natural language processing (NLP) system for automatic detection and classification of discrepancies in radiology reports.
• To enhance radiology resident education through structured, automated feedback.
• To analyze performance trends and variability among residents and attending radiologists.

Main Methods:

• Retrospective analysis of 889 lumbar spine MRI reports with preliminary and final versions.
• A multi-step NLP pipeline including sentence segmentation, BERT, GPT-4, and rule-based classification for 11 discrepancy types.
• Ground truth established by three radiologists; system performance evaluated using accuracy, sensitivity, specificity, and ICC.

Main Results:

• The NLP system demonstrated high accuracy (0.983-0.999), sensitivity (0.977-1.000), and specificity (0.900-1.000) across 11 correction types.
• Most common discrepancies were misdiagnosis (65.6%) and missed diagnosis (46.4%).
• Significant variability observed in resident error rates (e.g., missed diagnosis 11.1-59.1%) and attending radiologist correction patterns (p<0.001).

Conclusions:

• An NLP-based system can accurately identify and classify radiology report discrepancies, facilitating scalable and targeted resident feedback.
• Significant inter-resident and inter-attending variability underscores the need for individualized training programs.
• Standardized review practices are essential to improve consistency and quality in radiology reporting.