Performance Comparison of Cutting-Edge Large Language Models on the ACR In-Training Examination: An Update for 2025.

Area of Science:

• Artificial Intelligence in Medical Education
• Radiology Board Examination Preparation
• Large Language Model (LLM) Performance Evaluation

Background:

• Prior research evaluated large language model (LLM) performance on radiology board-style assessments.
• This study extends previous work by assessing newer LLMs on the ACR diagnostic radiology in-training examination (DXIT).

Purpose of the Study:

• To evaluate the performance of cutting-edge LLMs (GPT-4o, GPT-o1, GPT-o3, Claude, Gemini, Grok) on standardized DXIT questions.
• To compare model accuracy on text-based versus image-based questions to assess multi-modal reasoning.
• To investigate the impact of potential data contamination by comparing performance on original versus revised questions.

Main Methods:

• Seven LLMs were evaluated using 106 publicly available DXIT questions.
• Models were prompted using a standardized instruction set to simulate resident responses.
• Unadjusted and logic-adjusted accuracy were calculated, with subgroup analysis for text vs. image-based questions. Revised questions were used to test for data contamination.

Main Results:

• GPT-o1 (71.7%), GPT-4o (69.8%), and GPT-o3 (68.9%) achieved the highest unadjusted accuracy.
• Similar trends were observed for logic-adjusted accuracy, with GPT-o1, GPT-4o, and GPT-o3 outperforming other models.
• GPT-4o performed significantly better on text-based questions; performance on revised questions was comparable to original questions, suggesting limited data contamination.

Conclusions:

• Modern LLMs, particularly from OpenAI, demonstrate strong and improving performance on radiology board-style assessments.
• Comparable performance on revised prompts indicates a limited role for data contamination.
• LLMs show significant potential to support radiology resident education through personalized feedback and practice.