Related Experiment Videos
Large language model-assisted CAD-RADS classification: structured report extraction, accuracy, and clinical
Haorong Rong1, Xiyi Huang2,3, Xinjie Chen1
1Department of Radiology, The Eighth Affiliated Hospital, Southern Medical University, Foshan, Guangdong, China.
Objective:
To compare ChatGPT-4o, ChatGPT-5, and DeepSeek-R1 in assigning main Coronary Artery Disease Reporting and Data System (CAD-RADS) categories from findings-only coronary computed tomography angiography (CCTA) reports and to characterize management-relevant classification errors.
Methods:
This retrospective diagnostic-accuracy study included 2,100 category-balanced findings-only CCTA reports (300 per CAD-RADS category). Two experienced CCTA radiologists graded the English-translated reports; 12 disagreements were resolved by a third. ChatGPT-4o was evaluated in standard mode, ChatGPT-5 in Thinking mode, and DeepSeek-R1 in Deep Thinking mode; model generation and inference mode therefore varied together. Exact-match accuracy with 95% Wilson confidence intervals (CIs), paired comparisons, and unweighted Gwet AC1 with 95% CIs were calculated. Management-relevant transitions and digital subtraction angiography (DSA) findings among high-risk downgrades were analyzed.
Results:
DeepSeek-R1 had an exact-match accuracy of 82.57% (95% CI, 80.89-84.13) and AC1 of 0.797 (95% CI, 0.778-0.816). ChatGPT-5 had an accuracy of 80.19% (95% CI, 78.43-81.84) and AC1 of 0.769 (95% CI, 0.749-0.789), whereas ChatGPT-4o had an accuracy of 53.62% (95% CI, 51.48-55.74) and AC1 of 0.460 (95% CI, 0.435-0.485). Cochran's Q test was significant (Q = 657.87; df = 2; P < 0.001); after Holm correction, DeepSeek-R1 also outperformed ChatGPT-5 in the paired comparison (P = 0.026), and both outperformed ChatGPT-4o (both P < 0.001). ChatGPT-4o downgraded 127/900 (14.11%) reference high-risk reports to CAD-RADS 3 or lower. Of these 127 cases, 65 underwent DSA within three months; DSA supported the original high-risk assessment in 56/65 (86.15%). The corresponding high-risk downgrade proportions were 37/900 (4.11%) for ChatGPT-5 and 23/900 (2.56%) for DeepSeek-R1.
Conclusion:
The DeepSeek-R1 Deep Thinking and ChatGPT-5 Thinking configurations achieved higher accuracy than ChatGPT-4o in standard mode. Because model generation and inference mode varied together, the results do not isolate a model-family effect. LLMs merit further evaluation for secondary structuring when no documented CAD-RADS category is available, but should not overwrite a radiologist-assigned category or independently determine management.