Empowering front-line physicians with AI: Evaluating large language models in everyday ENT care

Area of Science:

• Medical Artificial Intelligence
• Otolaryngology
• Clinical Decision Support

Background:

• Large language models (LLMs) are emerging for clinical decision support, but their use in primary and emergency care, particularly for otolaryngology, is limited.
• Diagnostic uncertainty and inappropriate referrals in ear, nose, and throat conditions pose risks and inefficiencies.

Purpose of the Study:

• To compare the diagnostic, management, and referral performance of advanced LLMs against physicians in simulated otolaryngologic scenarios.
• To evaluate LLM capabilities in common and high-acuity ear, nose, and throat conditions relevant to primary and emergency care settings.

Main Methods:

• Twelve validated otolaryngology clinical vignettes were used.
• One hundred physicians and four LLMs (Gemini-2.0, ChatGPT-4.0, ChatGPT-5, OpenEvidence) completed the vignettes.
• Outputs were rated by a blinded expert panel using the Quality Analysis of Medical Artificial Intelligence tool.

Main Results:

• Physicians achieved 91.6% diagnostic and 87.9% management accuracy.
• LLMs showed comparable diagnostic and management accuracy to physicians.
• LLMs demonstrated higher referral appropriateness compared to physicians, who had a 30.4% inappropriate referral rate in non-urgent cases.

Conclusions:

• LLMs exhibit consistent, guideline-concordant reasoning in simulated otolaryngology cases.
• LLMs have potential to support, not replace, clinical judgment in emergency and primary care.
• Responsible integration and real-world validation are crucial for LLM adoption in clinical practice.