RadCLARE: an automated clinical language engine for detecting semantic errors in radiology reports

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing in Healthcare
• Radiology Report Analysis

Background:

• Radiology report errors can lead to incorrect clinical decisions.
• Automated systems are needed to improve the accuracy of radiology reports.
• Investigating the potential of large language models (LLMs) to mitigate these errors is crucial.

Purpose of the Study:

• To develop and evaluate an automated engine, RadCLARE, for detecting semantic errors in Chinese radiology reports.
• To assess the impact of RadCLARE on the overall error rate in radiology reporting.
• To gauge radiologist satisfaction with the implemented system.

Main Methods:

• Developed RadCLARE, a BERT-based engine, trained on 1.4 million Chinese radiology reports (DR, CT, MR).
• Utilized expert manual annotation for 1,000 reports to establish a reference standard.
• Compared RadCLARE's performance against expert annotations and analyzed error rate changes before and after implementation.

Main Results:

• RadCLARE achieved high performance: 87.3% accuracy, 88.3% precision, 86.4% recall, and 87.4% F1-score in detecting semantic errors.
• The semantic error rate in radiology reports significantly decreased from 4.19% in the baseline dataset to 0.85% after RadCLARE implementation (p < 0.001).
• 95.7% of participating radiologists expressed satisfaction with the RadCLARE system.

Conclusions:

• The RadCLARE network effectively detects semantic errors in radiology reports with high accuracy.
• Implementation of RadCLARE led to a substantial reduction in the semantic error rate.
• The system shows promise for improving radiology report quality and reducing clinical decision errors.