The inter-rater reliability of the TEDNQ for evaluating emergency department note quality

Audits are a strategy for assessing and improving the quality of Emergency Department documentation. Based on expert input, the TEDNQ (Tool for Emergency Department Note Quality) was recently developed for this purpose. This study assessed the inter-rater reliability of the TEDNQ and its correlation with global impressions of Emergency Department note quality.

Four emergency physicians received standardized training and then evaluated 100 notes from visits for patients with abdominal pain presenting to a Canadian academic Emergency Department. Raters provided TEDNQ and global impression scores. Inter-rater reliability was calculated using Fleiss' kappa for global scores and intraclass correlation coefficient (ICC) for TEDNQ scores. Point biserial correlations between TEDNQ and global impressions for the set of notes were calculated.

Across all raters, the ICC for the total TEDNQ scores was 0.40 (95% CI 0.30-0.50), 0.40 (95% CI 0.30-0.51) for the universal component, and 0.23 (95% CI 0.13-0.34) for the conditional components. The Fleiss' kappa for global impressions across all pairs of reviewers was 0.18 (95% CI 0.10-0.26). The point biserial correlation between TEDNQ and global scores was 0.44 (p < 0.001).

While validated tools to assess documentation quality are essential, the recently developed TEDNQ is not yet suitable for widespread use by emergency physicians. Global impressions of note quality also demonstrate limited usefulness due to poor inter-rater reliability. Further refinement and testing of the TEDNQ is warranted prior to prospective evaluation across diverse emergency department settings.