Assessment of frontier Large Language Models in sleep medicine

Area of Science:

• Artificial Intelligence in Medicine
• Medical Diagnostics
• Sleep Medicine

Background:

• Large language models (LLMs) are increasingly evaluated for medical applications.
• Frontier LLMs show promise but require rigorous assessment in specialized domains.
• Sleep medicine presents unique diagnostic reasoning challenges.

Purpose of the Study:

• To compare the diagnostic reasoning and knowledge recall capabilities of ChatGPT-5 and Grok-4 in sleep medicine.
• To assess LLM performance on clinical vignettes and multiple-choice questions relevant to sleep disorders.

Main Methods:

• Two LLMs, ChatGPT-5 and Grok-4, were tested on 79 sleep medicine case vignettes and 897 multiple-choice questions.
• Performance metrics included exact match for diagnosis, F1-score for differential diagnosis, and proportion correct for MCQs.
• Statistical comparison used the Mann-Whitney U test.

Main Results:

• Both models achieved high accuracy for final diagnosis (92.4%) and MCQs (ChatGPT-5: 93.0%, Grok-4: 92.8%).
• Differential diagnosis generation yielded modest F1-scores (ChatGPT-5: 0.55, Grok-4: 0.59).
• No statistically significant performance differences were observed between the two LLMs.

Conclusions:

• Frontier LLMs demonstrate strong knowledge recall but limited complex clinical reasoning in sleep medicine.
• Current general-purpose LLMs are more suitable for focused knowledge support than broad diagnostic hypothesis generation.
• Future research should explore domain-adapted models and clinician-in-the-loop systems to enhance LLM utility in clinical practice.