Artificial Intelligence in Radiology: Performance of ChatGPT-4v and GPT-4o on Diagnostic Radiology in-Training (DXIT)...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Large Language Models in Radiology

Background:

• The Diagnostic Radiology in-Training (DXIT) examination assesses radiology resident knowledge.
• Large language models (LLMs) are increasingly explored for medical education and clinical support.

Purpose of the Study:

• To evaluate the performance of GPT-4v and GPT-4o on the DXIT examination.
• To compare AI performance on image-based versus text-only radiology questions.

Main Methods:

• 1,136 DXIT questions were administered to GPT-4v and GPT-4o.
• Models provided answers, rationales, and confidence levels.
• Accuracy was analyzed using statistical tests and receiver operating characteristic curves.

Main Results:

• GPT-4o (73.5%) and GPT-4v (69.3%) outperformed the national average resident score (61.9%).
• Both models struggled with image-based questions (55.6% and 50.3% accuracy).
• AI confidence levels poorly predicted answer correctness (AUC 0.64-0.66).

Conclusions:

• GPT-4o demonstrated superior performance over GPT-4v across most metrics.
• Despite overall success, AI performance on image-based questions lags behind text-only questions and human trainees.
• Caution is advised when using these AI models for radiology test preparation or image interpretation due to limited reliability.