Large Language Models Solving the European Diploma in Radiology: A Comparative Evaluation

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology Education Technology
• Large Language Models (LLMs)

Background:

• Multimodal large language models (LLMs) are rapidly advancing, showing significant potential in radiology education and assessment.
• Evaluating the performance of these LLMs in standardized medical examinations is crucial for understanding their capabilities and limitations.

Purpose of the Study:

• To compare the performance of three state-of-the-art LLMs: GPT-5, Gemini 2.5 Pro, and Claude 4.5 Sonnet.
• To assess their capabilities using a retired European Diploma in Radiology (EDiR) examination.

Main Methods:

• The EDiR examination, including Multiple Response Questions, Short Cases, and CORE cases, was administered to LLMs under zero-shot conditions.
• Inputs comprised text, static images, and videos; responses were scored against official European Board of Radiology criteria.
• Passing thresholds were based on historical human performance data.

Main Results:

• Gemini 2.5 Pro (72.6%) and GPT-5 (67.3%) surpassed the written component pass mark (50.9%), while Claude 4.5 Sonnet scored 47.5%.
• GPT-5 (62.5%) and Gemini 2.5 Pro (56.3%) passed the CORE component (pass mark 55%), unlike Claude 4.5 Sonnet (50.7%).
• Models excelled in text and static image tasks but struggled with specific coordinate localization and dynamic video interpretation.

Conclusions:

• GPT-5 and Gemini 2.5 Pro met EDiR passing criteria, indicating suitability for augmenting radiology education.
• Despite advancements in semantic processing, limitations in spatial localization and temporal reasoning persist, hindering autonomous clinical deployment.
• Clinical visual grounding remains a key challenge for current LLMs in radiology.