Error Detection in Emergency Radiology Reports Using a Large Language Model: Multistage Evaluation Study

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging and Diagnostics
• Radiology Informatics

Background:

• Emergency radiology faces challenges with increasing workloads and the risk of reporting errors.
• The efficacy of large language models (LLMs) in identifying errors in emergency radiology, particularly in non-English contexts, is not well-established.
• Accurate and timely reporting is critical in emergency radiology.

Purpose of the Study:

• To evaluate the performance of a domain-optimized LLM, DeepSeek-R1, in detecting errors within Chinese emergency radiology reports.
• To compare the error detection capabilities of DeepSeek-R1 against board-certified radiologists.
• To assess the potential of DeepSeek-R1 as an assistive tool for quality control in emergency radiology.

Main Methods:

• A dataset of 7435 Chinese emergency radiology reports was compiled.
• Five LLMs were initially screened, with DeepSeek-R1 selected for further evaluation using 0-shot and few-shot learning techniques.
• Model performance was benchmarked against 12 radiologists and validated on real-world reports.

Main Results:

• DeepSeek-R1 demonstrated a higher error detection rate (84.4%) in the few-shot setting compared to the 0-shot setting (60.9%).
• The LLM outperformed radiology residents and showed comparable performance to senior and attending radiologists.
• DeepSeek-R1 identified critical omissions and other errors more effectively than residents and operated with greater efficiency.

Conclusions:

• The domain-optimized LLM, DeepSeek-R1, shows significant potential for improving the quality control of emergency radiology reports.
• Its performance and efficiency suggest its utility as a valuable assistive proofreading tool in clinical radiology workflows.
• DeepSeek-R1 can aid in reducing errors and enhancing the accuracy of diagnostic reporting under time constraints.