Comparative Analysis of ChatGPT-4o and Gemini Advanced Performance on Diagnostic Radiology In-Training Exams

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Radiology Diagnostics

Background:

• Growing integration of AI in medical education and clinical practice.
• Increasing interest in large language models (LLMs) for diagnostic reasoning and training.
• Uncertainty regarding LLM accuracy in analyzing both written and image-based radiology content.

Purpose of the Study:

• Evaluate the performance of ChatGPT-4o and Gemini Advanced on the 2022 ACR Diagnostic Radiology In-Training (DXIT) Exam.
• Assess LLM capabilities across different radiological subfields.
• Compare AI model performance in interpreting written versus image-based radiology questions.

Main Methods:

• Testing ChatGPT-4o and Gemini Advanced on 106 multiple-choice questions from the 2022 DXIT exam.
• Including both image-based and written-based questions across various radiological subspecialties.
• Comparing performance using overall accuracy, subfield-specific accuracy, and two-proportion z-tests.

Main Results:

• ChatGPT-4o achieved 69.8% overall accuracy, outperforming Gemini Advanced (60.4%), though not statistically significant.
• ChatGPT-4o showed higher accuracy on image-based questions (57.8%) compared to Gemini Advanced (43.8%).
• Both models demonstrated similar accuracy on written-based questions (88.1% vs. 85.7%) and varied performance across subfields.

Conclusions:

• LLMs show promise for radiology education and text-based assessments.
• Limitations in image interpretation accuracy and response consistency require further AI model refinement.
• Future research should focus on enhancing AI reliability, multimodal capabilities, and integration into radiology training.