Large Language Models as Raters of Hospital Incident Reports: A Blinded Multi-Model Evaluation of Severity...

Large language models (LLMs) are increasingly proposed to classify hospital incident reports, but evidence is dominated by single-model studies reported as aggregate concordance. The authors evaluated whether frontier LLMs reliably classify incident reports and grade harm severity, whether independent models err in correlated ways, and which failure modes matter clinically.

Five LLMs (ChatGPT, Gemini, Claude, DeepSeek, Mistral) classified 824 anonymized incident reports from free text alone, using an identical prompt through providers' standard commercial interfaces, blinded to the reference and one another. Outputs were compared with an expert-adjudicated reference for event type and severity, with the original reporter, and across models (agreement statistics, calibration, severe event detection, ensembling, and a sentinel code audit). A 150-report subset was readministered > 2 months later for test-retest reliability; criterion-based and few-shot prompts tested prompt dependence.

Models classified event type reliably (κ 0.82-0.84) but graded severity poorly (weighted κ 0.39-0.47). Of 140 severe events, sensitivity was 0.41-0.50, rising to 0.68-0.79 when harm was documented in the narrative; much apparent failure reflected information insufficiency and construct mismatch. Errors were correlated (mean model-model κ exceeded model-reference κ), majority voting did not help, and no model surpassed the reporter. Confidence was weakly informative and sentinel codes often inconsistent with official content. On readministration, exact severity agreement was 0.59-0.80, with severe-to-no/mild-harm transitions in four of five models; explicit definitions or few-shot examples did not remove it in four of five models.

Aggregate agreement concealed failure concentrated in severe events and shared across models. In this one-shot, model-only setting, LLMs classified event type reliably but were not adequate for autonomous severity or sentinel event grading; these determinations should remain with expert adjudication under algorithmovigilance.