Development and Real-World Evaluation of an NLP-Assisted Quality Control System for Radiology Reports

Maintaining the quality and consistency of radiology reports has become increasingly challenging with the growing volume of imaging examinations. This study aimed to develop and evaluate an NLP-assisted quality control system for routine radiology reports in a real-world clinical setting. An NLP-assisted quality control module was integrated into the HIS-PACS workflow at Tangdu Hospital for real-time analysis of free-text radiology reports. A total of 215,870 reports generated between August 1, 2024, and February 28, 2025, were retrospectively analyzed. Report error rates were compared across physician grades, examination locations, device types, and working periods, and temporal trends were further assessed using ARIMA time-series modeling. Significant differences in report error rates were found among different physician grades, particularly in description, unit, orientation, and organ resection errors. Error rates also varied significantly by examination location, device type, and time of day (morning, afternoon, evening). The ARIMA model predicted a declining trend in error rates over the next 3 months (respectively 0.036, 0.031, and 0.029). The NLP-assisted system can identify potential reporting errors and provide real-time quality control support for radiology reports. Its implementation may help improve reporting consistency and support workflow efficiency.