Guideline-Integrated Large Language Models Improve Decision Support for Acute Ear, Nose and Throat Emergencies

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Otolaryngology

Background:

• Acute otolaryngologic emergencies require timely, guideline-concordant decisions in emergency departments (EDs).
• Large language models (LLMs) show potential for clinical decision support but struggle with guideline adherence in time-sensitive situations.
• Retrieval-augmented generation (RAG) may enhance LLM accuracy by integrating authoritative guideline content.

Purpose of the Study:

• To evaluate if RAG integration of American Academy of Otolaryngology-Head and Neck Surgery (AAOHNS) guidelines improves LLM decision support for acute ear, nose, and throat (ENT) emergencies.
• Assess improvements in accuracy, guideline adherence, and safety of LLM outputs.

Main Methods:

• Developed 12 standardized clinical vignettes for sudden sensorineural hearing loss, epistaxis, and Bell's palsy.
• Two LLMs (ChatGPT, Gemini) generated responses with and without RAG-integrated guidelines.
• Five blinded otolaryngologists evaluated outputs using a rubric assessing diagnostic accuracy, workup, management, referral, guideline adherence, and safety.

Main Results:

• RAG-enabled LLMs showed significant improvements in diagnostic accuracy, workup quality, management planning, and referral appropriateness.
• Guideline adherence was significantly enhanced with RAG.
• RAG reduced misleading recommendations and modestly decreased overtreatment; overtesting rates remained unchanged.

Conclusions:

• Guideline integration via RAG enhances the reliability, safety, and guideline alignment of LLM outputs for acute otolaryngologic emergencies.
• RAG-supported LLMs show promise for improving evidence-based decision-making in emergency care settings.