Can we tell when an error is an error? Assessing the reliability of a Likert scale in measuring error

Error is often difficult to define. The objective of this study was to determine reliability of EM physician quality reviews in detecting medical error. Retrospective data were collected in an academic ED, with an annual census of 57,000, from 11/2012-11/2024. Cases reviewed were 1) returns to ED <72 h and admitted; 2) admissions from the ED to the floor and transferred to the ICU <24 h; 3) expired <24 h of ED arrival; 4) referred for QA review from provider or patient complaints. All cases were randomly assigned to a QA-trained EM physician not involved in the patient's care. Reviewers used an 8-point Likert scale to assess whether an error occurred, categorized as either definite, possible, judgment call, or no error. To help quantify level of agreement, random cases were assigned to at least 2 reviewers, with agreement assessed using Cohen's Kappa. Reviewer agreement was categorized into complete agreement, slight discordance, and higher levels of discordance with and without weights. Of 19,200 reviews, 1845 had 2 reviewers (9.6 %). There was moderate agreement when using a Likert scale with weights, (Kappa = 0.43[0.39-0.44]) and a grouped scale with weights, Kappa of 0.44 (0.38-0.47 when Likert ratings were grouped into no error (1), possible (2), probable(3), and definite (4-8). When comparing complete agreement between reviewer as to the likelihood of error, agreement diminished. When reviewers agreed on the presence of an error, they disagreed on its severity, both in weighted and unweighted analyses. Reviewers tended to agree when an error was an error. These findings underscore the need for standardized error classification frameworks to ensure reliability of reviews.