Radiology Board-Style Examinations and Large Language Models: A Scoping Review of Model Performance.

Area of Science:

• Artificial Intelligence in Medical Education
• Radiology Examination Assessment
• Natural Language Processing in Healthcare

Background:

• Large Language Models (LLMs) are being assessed for their capability to answer radiology board-style questions.
• Understanding LLM accuracy and limitations is crucial for their educational applications in radiology.

Purpose of the Study:

• To review the current literature on LLM performance in radiology board-style examinations.
• To identify trends in LLM evaluation, including model types, modalities, and outcomes.

Main Methods:

• A scoping review was performed in October 2025, adhering to PRISMA guidelines.
• Searches were conducted in PubMed, Scopus, and Web of Science, including 29 studies evaluating LLMs on radiology board questions.

Main Results:

• GPT-4 and its variants demonstrated high accuracy (≈65-90%) in text-based tasks, sometimes surpassing human performance.
• Multimodal tasks showed lower accuracy (45-89%), and open-source models achieved comparable results to proprietary ones.
• Direct comparisons between LLM and human radiologist performance were limited in the reviewed studies.

Conclusions:

• LLMs, particularly GPT-4 based models, show significant potential for answering text-based radiology board-style questions.
• Limitations exist in multimodal applications and complex reasoning, requiring further research and development.