Performance of large language models on sleep medicine certification examination: a comprehensive multi-model...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Natural Language Processing

Background:

• Contemporary large language models (LLMs) show promise in various professional domains.
• Evaluating LLM performance on specialized medical knowledge is crucial for their safe and effective integration.

Purpose of the Study:

• To compare the performance of nine LLM configurations on sleep medicine certification examination-aligned questions.
• To analyze differences based on LLM versions, pricing tiers, and subdomain competencies.

Main Methods:

• A cross-sectional comparative analysis used 197 multiple-choice questions aligned with American Academy of Sleep Medicine (AASM) standards.
• Nine LLMs were tested, including ChatGPT, Gemini, Claude, Deepseek V3, xAI Grok3, and Llama 3, with multiple iterations per question.
• Performance was assessed using a strict 3/3 concordance criterion, with supplementary analyses on majority voting.

Main Results:

• Significant performance heterogeneity was observed across LLMs, with accuracy ranging from 68.5% to 95.9%.
• Gemini 2.5 Pro (95.9%), Claude Opus 4 (93.9%), and ChatGPT GPT-4o (93.4%) achieved the highest accuracy.
• Premium LLM versions consistently outperformed free alternatives, and performance varied across sleep medicine subdomains.

Conclusions:

• Contemporary LLMs, particularly premium versions, demonstrate high accuracy on certification-style questions, exceeding the 80% benchmark.
• The superior performance of paid models highlights considerations for equitable access to AI-driven medical education and clinical tools.
• Results are based on a question bank and do not replicate official board examination conditions.