Observer agreement in the grading of oral epithelial dysplasia

To determine the extent of observer agreement in diagnosis of oral epithelial dysplasia (OED). Published studies of OED examiner agreement report relatively low agreement levels; however, these studies were limited by the methodologies employed.

For this study, 64 slides were each independently examined twice by three oral pathologists. Consistency was assessed by determining intra- and interexaminer agreement. Conformity was assessed by using the modal diagnosis as a gold standard.

The group showed moderate interobserver agreement when grading the presence or absence of OED with a group-simple kappa (Ks) of 0.51 (95% CI = 0.42-0.61), and substantial agreement when using a 5-point ordinal scale with a group-weighted kappa (Kw) of 0.74 (95% CI = 0.64-0.85). The group showed fair to substantial intraexaminer agreement when assessing the presence or absence of OED, with Ks ranging from 0.22 to 0.78, and showing almost a perfect agreement using a 5-point ordinal scale, with Kw ranging from 0.82-0.96. Conformity with the comparison standard modal diagnosis was almost perfect, with pairwise Kw ranging from 0.81 to 0.92.

Overall, there was substantial intra- and interobserver consistency and almost perfect conformity in the grading of OED. Appropriate statistical methods are necessary to determine the degree of observer agreement.