Performance of ChatGPT and Large Language Models on Medical Licensing Exams Worldwide: A Systematic Review and...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Education Technology

Background:

• Large language models (LLMs) are increasingly evaluated on medical licensing exams.
• Existing research on LLM performance is fragmented across different models, exam systems, and languages.
• A systematic assessment is needed to understand LLM capabilities in this domain.

Purpose of the Study:

• To systematically assess and compare the performance of various large language models (LLMs) on multiple medical licensing examinations.
• To provide pooled estimates and network meta-analysis of LLM accuracy across different languages and exam systems.
• To identify factors influencing LLM performance, such as model version, exam system, and language.

Main Methods:

• Conducted a meta-analysis of 120 evaluations from 10 exam systems in nine languages (2021-June 2025).
• Utilized comprehensive literature searches across PubMed, Web of Science, and IEEE Xplore.
• Employed random-effects meta-analysis, network meta-analysis, and moderator-aware meta-regression.

Main Results:

• 13 out of 16 evaluated LLMs exceeded the 60% passing threshold.
• GPT-o1 achieved the highest accuracy (95.4%), followed by DeepSeek-R1 (92.0%) and GPT-4o (89.4%).
• Performance varied significantly by model, exam system, and language, with lower accuracy in Chinese/Japanese and higher in German/Peruvian exams.

Conclusions:

• Several LLMs now demonstrate accuracy comparable to or exceeding passing standards for medical licensing exams.
• LLMs show significant potential for application in medical education and clinical decision support.
• Model type, exam system, and language are key determinants of LLM performance heterogeneity.