Diagnostic performance of Large Language Models (LLMs) compared with physicians in sleep medicine

Area of Science:

• Medical Artificial Intelligence
• Clinical Diagnostics
• Sleep Medicine Research

Background:

• Artificial intelligence (AI), specifically large language models (LLMs), is being explored for medical diagnostic applications.
• LLMs may enhance clinicians' diagnostic reasoning when integrated into clinical systems.
• The diagnostic performance of LLMs in sleep medicine has not been evaluated against expert physicians.

Purpose of the Study:

• To compare the diagnostic accuracy of three prominent LLMs against experienced sleep physicians.
• The study utilized real-world clinical case vignettes for evaluation.

Main Methods:

• Sixteen diverse sleep disorder vignettes from the AASM Case Book (2019) were used.
• Three LLMs (ChatGPT-4, Gemini 2.0, DeepSeek) and three board-certified sleep physicians independently assessed the vignettes.
• Differential diagnoses were compared to AASM reference lists, and final diagnoses were scored using a 3-point Likert scale.

Main Results:

• LLMs showed comparable mean agreement percentages for differential diagnoses (70.7%-77.7%) to physicians (72.9%).
• No statistically significant difference was found in differential diagnostic accuracy between LLMs and physicians (p=0.839).
• LLMs achieved an average final diagnosis concordance score of 87.5%, within the range of expert physicians (81.3%-96.9%).

Conclusions:

• LLMs exhibit diagnostic performance comparable to experienced sleep clinicians.
• LLMs show potential as supplementary tools to aid in medical diagnosis.
• Further research is recommended to explore broader applications and integration of LLMs in clinical practice.