Related Experiment Videos
A Hybrid Human-AI Workflow for Ultrasound Report Quality Control: A Multicenter Performance and Efficiency Evaluation
Xuan Liang1, Hui He2, Chunzheng Liu3
1Department of Diagnostic and Therapeutic Ultrasonography, Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Center for Cancer; Tianjin's Clinical Research Center for Cancer; Key Laboratory of Cancer Prevention and Therapy, Tianjin, China (X.L., J.Z., X.W., W.H., D.L., J.W., and X.W.).
Rationale And Objectives:
This study evaluated a hybrid human-AI workflow for ultrasound report quality assurance using multicenter retrospective data.
Materials And Methods:
786 anonymized ultrasound reports covering 22 examination categories were collected from four hospitals. Reports containing naturally occurring or implanted errors were matched across arms. GPT-4o and DeepSeek-v3 performed initial screening before review by six human reviewers (HRs). HR with blinded expert over-reading (HR-ex) served as control. Accuracy, false-positive (FP) and false-negative (FN) rates, and review time were compared. Supplementary validation compared HR and DeepSeek-assisted review using 1680 consecutive reports without implanted errors.
Results:
Hybrid workflow (HYB) accuracy was comparable to HR (4o-HYB 80.2%, v3-HYB 82.5%, HR 82.3%) but lower than HR-ex (87.3%). Both HYBs showed higher FN rates than HR-ex (4o-HYB 34.2% and v3-HYB 27.3% vs HR-ex 16.9%, both p < 0.001). Severity assessment did not indicate preferential omission of clinically significant errors. Human verification reduced LLM-generated FNs and FPs. Median review time decreased substantially from 51.0 s (HR-ex) to 11.0 s (4o-HYB) and 12.0 s (v3-HYB). In supplementary validation (error prevalence, 9.6%), v3.2-HYB and HR showed FN rates of 22.6% and 24.3%, FP rates of 7.7% and 10.0%, and median review times of 15.4 and 24.3 s, respectively.
Conclusion:
The LLM-assisted HYB achieved accuracy comparable to single-physician review while substantially reducing audit time. These findings require prospective real-world validation before routine clinical implementation.