Large Language Models Versus Human Readers in CAD-RADS 2.0 Categorization of Coronary CT Angiography Reports

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology Reporting Systems
• Cardiovascular Disease Assessment

Background:

• Accurate classification of coronary artery disease (CAD) is crucial for patient management.
• Coronary CT angiography (CCTA) reports are essential for CAD diagnosis.
• Large Language Models (LLMs) show potential in automating medical report analysis.

Purpose of the Study:

• To evaluate the accuracy of LLMs in assigning Coronary Artery Disease Reporting and Data System (CAD-RADS) 2.0 categories and modifiers.
• To compare the performance of LLMs against human readers using real-world CCTA reports.
• To assess the processing time efficiency of LLMs compared to human interpretation.

Main Methods:

• 180 CCTA reports were randomly selected and independently assessed by four LLMs (O1, GPT-4o, GPT-4, GPT-3.5-turbo) and four human readers.
• A consensus panel of two expert cardiac radiologists established the reference standard for CAD-RADS 2.0 categorization.
• LLMs received CCTA reports and CAD-RADS 2.0 summaries as input prompts; accuracy was compared using McNemar tests.

Main Results:

• O1 achieved the highest accuracy (90.7%) in full CAD-RADS categorization, outperforming other LLMs and most human readers.
• O1's accuracy was comparable to experienced cardiac radiologists and showed no significant difference compared to two residents.
• LLMs processed reports significantly faster (1.34-16.61s) than human readers (32.10-55.06s); O1 achieved 95.7% accuracy in external validation.

Conclusions:

• The LLM O1 demonstrates high accuracy and efficiency in assigning CAD-RADS 2.0 categories from CCTA reports.
• O1 offers comparable or superior performance to human readers with substantially reduced processing times.
• O1 represents a promising tool for enhancing the efficiency and consistency of CAD-RADS reporting in clinical practice.