Study design for calibration of clinical examiners measuring periodontal parameters

We present an approach to examiner calibration study design where the number of calibration subjects is based on a specified margin of error (half-width of the 95% confidence interval [CI]) of the percentage of agreement (exact and within 1 mm) for both intra- and interexaminer reliability assessments.

An experienced standard examiner (S) trained three dental hygienists (A, B, and C) in correct procedures for obtaining a variety of periodontal measures. Duplicate measurements of probing depth (PD [mm]) and the free gingival margin to the cemento-enamel junction (CEJ-GM [mm]) were obtained in a pilot study to design a formal examiner calibration study, where sample sizes were adjusted for the effects of within-subject clustering of binary indices of agreement.

Within-subject clustering of agreement indices resulted in an approximate four-fold increase in the variance of the estimates of percentage of agreement with the standard. PD and CEJ-GM percentage of exact agreement measurements (95% CI) for each examiner-standard pair, respectively, were as follows: AS=55% (48%, 61%) and 70% (62%, 78%); BS=52% (45%, 59%) and 73% (63%, 82%); and CS=55% (50%, 61%) and 72% (65%, 79%). The corresponding 95% CIs unadjusted for the effects of clustering underestimated the margin of error associated with the estimates of exact agreement by as much as 57% for PD and 68% for CEJ-GM.

Failure to account for dependence among site-level agreement indices results in a false sense of precision in the resulting reliability estimates and can lead to faulty inference.