Performance of Large Language Models on Diagnostic Radiology Board-Style Questions: A Comparative Evaluation of...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Radiology Informatics

Background:

• Large language models (LLMs) are increasingly used in various fields, including medicine.
• Evaluating the diagnostic accuracy and internal consistency of LLMs for specialized medical examinations is crucial.
• Radiology board questions require high diagnostic precision and consistency.

Purpose of the Study:

• To compare the diagnostic accuracy and internal consistency of GPT-4o, Perplexity AI, and OpenEvidence on text-based radiology board questions.
• To assess the performance of these LLMs across different radiology subspecialties.
• To determine if specialized AI models offer improved reliability over general-purpose models in a medical context.

Main Methods:

• 161 text-based multiple-choice questions from the American College of Radiology (ACR) Diagnostic Radiology In-Training Examination were used.
• Each LLM (GPT-4o, Perplexity AI, OpenEvidence) was run three times independently.
• Diagnostic agreement was assessed using Cohen's kappa, with performance analyzed by subspecialty.

Main Results:

• Perplexity AI achieved the highest agreement with the ACR reference key (κ=0.883), followed by OpenEvidence (κ=0.858) and GPT-4o (κ=0.709).
• All models demonstrated high internal consistency, with OpenEvidence showing absolute consistency (κ=1.00).
• Perplexity AI exhibited the least performance variability across 14 radiology subspecialties.

Conclusions:

• Emerging LLMs like Perplexity AI and OpenEvidence show promising diagnostic reliability for radiology-specific applications.
• Specialized AI models may outperform general-purpose LLMs in accuracy and consistency for medical board examinations.
• Further research into AI applications in medical education and diagnostics is warranted.