Automated auditing of emergency department documentation using large language models

Errors in emergency department (ED) documentation can lead to patient harm and medicolegal risk, however manual document auditing is resource-intensive and difficult to scale. Large language models (LLMs) may offer an automated alternative, but their performance for task-specific error detection remains uncertain. This study evaluated four state-of-the-art LLMs for automated auditing of ED documentation.

A total of 600 ED discharge notes from real-world encounters were evaluated, including 500 notes with a single injected documentation error across five predefined categories (patient/encounter, medication/allergy, clinical findings, investigation failure/contradiction, or disposition/follow-up), and 100 error-free reference notes. Four LLMs (Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5, and Grok 4) independently audited each note under blinded conditions. Model performance was assessed using accuracy, precision, recall, and F1 score, with pairwise comparisons performed using McNemar's test.

Model accuracies ranged from 93.2% to 94.2% for detection of any documentation error, with precision exceeding 98% and F1 scores above 0.95 for all models. Claude Sonnet 4.5 achieved the highest overall accuracy (94.2%) and F1 score (0.964). Pairwise comparisons showed no statistically significant differences in overall error detection between models (p > 0.05). At the category level, all models demonstrated near-perfect performance for patient/encounter inconsistencies and medication/allergy errors (F1 scores >0.96), with lower performance for investigation failures/contradictions, followed by disposition/follow-up errors and clinical finding inconsistencies.

LLMs demonstrate strong performance in auditing ED documentation, supporting their potential role as quality assurance tools in emergency care. Future work should explore how these systems can be integrated into real clinical workflows.