Benchmarking LLM decision support in inflammatory aneurysms: DeepSeek-R1, DeepSeek-V3 and ChatGPT-4o

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Vascular Surgery Decision Support

Background:

• Guideline-concordant decision support is crucial for managing inflammatory aneurysms.
• Evaluating the accuracy and efficiency of large language models (LLMs) in this critical domain is necessary.

Purpose of the Study:

• To compare the diagnostic, interventional, and surveillance decision-making accuracy and time efficiency of three LLMs: DeepSeek-R1, ChatGPT-4o, and DeepSeek-V3.
• To assess LLM performance against contemporary aortic guidance.

Main Methods:

• A prospective, rater-blinded evaluation using a 50-item instrument aligned with aortic guidelines.
• Five vascular specialists scored LLM outputs using a 0/1/2 rubric.
• Analysis included total accuracy, completion time, and domain-specific subscores, with inter-rater reliability assessed.

Main Results:

• Mean total accuracy scores were 89.8% (DeepSeek-R1), 88.4% (ChatGPT-4o), and 77.8% (DeepSeek-V3), with R1 and 4o showing near-parity and significantly outperforming V3.
• Completion times varied significantly: DeepSeek-V3 (19.8s), ChatGPT-4o (33.4s), and DeepSeek-R1 (58.5s).
• Accuracy correlated positively with completion time (Pearson r=0.802), with differences concentrated in guideline-derived items.

Conclusions:

• In complex, safety-critical decisions for inflammatory aneurysms, slower, reasoning-intensive LLMs achieve higher accuracy.
• A low-latency multimodal system (ChatGPT-4o) provides near-parity accuracy at significantly reduced times.
• Model selection should align with task criticality, and human oversight remains essential.