Comparative Performance of Seven Mainstream Large Language Models on the 2022 American College of Radiology...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging and Radiology
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in medical education, but comparisons on radiology-specific assessments are limited.
• Recent LLMs like Grok 4.1, Bing Copilot GPT-5, DeepSeek V3.2, and OpenEvidence have not been evaluated on the ACR DXIT examination.

Purpose of the Study:

• To compare the performance of seven contemporary LLMs on the 2022 American College of Radiology Diagnostic Imaging In-Training (ACR DXIT) examination.
• To stratify LLM performance by question format (written-only vs. image-based) and radiology subject domain.

Main Methods:

• Seven LLMs were tested on 106 multiple-choice questions from the 2022 ACR DXIT exam.
• Five multimodal models and two text-only models were used, with a standardized prompt.
• Statistical analysis included Cochran's Q test and McNemar's test, with 95% confidence intervals calculated.

Main Results:

• Multimodal LLMs achieved 65.1%-76.4% accuracy, with no significant differences between models.
• All multimodal models performed significantly better on written (88.1%-95.2%) than image-based (46.9%-64.1%) questions.
• Strengths were noted in ultrasound and chest radiology, while musculoskeletal imaging showed weakness.

Conclusions:

• Contemporary multimodal LLMs demonstrate moderate accuracy on radiology in-training exams, surpassing previous benchmarks.
• A persistent performance gap exists between written and image-based questions, indicating limitations in radiologic image interpretation.
• Current LLMs may aid radiology education for non-interpretive content but are unsuitable for visual diagnostic reasoning.