Artificial intelligence in sleep medicine: assessing the diagnostic precision of ChatGPT-4

Large language models such as ChatGPT-4 are emerging in medicine, including sleep medicine, where artificial intelligence is used to analyze sleep data and predict treatment outcomes. Effectiveness of large language models in accurately diagnosing sleep disorders based on clinical history has not yet been studied. This study evaluates ChatGPT-4's diagnostic performance using clinical vignettes.

Nineteen clinical vignettes containing patient history, physical examination findings, and diagnostic tests from the Case Book of Sleep Medicine (third edition, 2019, American Academy of Sleep Medicine) were presented to ChatGPT-4. Its differential and final diagnoses were compared to reference diagnoses, with accuracy assessed by (1) the percentage of correct differentials and (2) a 3-tier scoring system (no match, partial match, full match) for final diagnoses.

The mean accuracy for differential diagnoses was 63.27% ± 15.61% (standard deviation), ranging from 33.33-100%. The mean number of artificial intelligence-generated differential diagnoses matching the American Academy of Sleep Medicine case differential diagnoses was 2.79 ± 0.71 (standard deviation). For final diagnoses, ChatGPT-4 scored a total of 30 out of a possible 38, resulting in an overall accuracy of 78.95%. The model achieved a mean score of 1.58 ± 0.61 (standard deviation) out of 2, with 68.42% of cases achieving a full match. Performance was higher in cases with fewer differential diagnoses, whereas accuracy decreased in complex cases.

ChatGPT-4 demonstrates promising diagnostic potential in sleep medicine, with moderate to high accuracy in identifying differential and final diagnoses, although its variability in more complex cases calls for refinement and clinical validation.

Patel A, Cheung J. Artificial intelligence in sleep medicine: assessing the diagnostic precision of ChatGPT-4. J Clin Sleep Med. 2025;21(9):1511-1517.