Agreement in Scoring Preclinical Dental Procedures: Impact on Grades and Instructor-Related Determinants

The aims of this study were to assess agreement among instructors in scoring preclinical dental procedures, the effect of this agreement on grades, and factors associated with agreement and grades differences. Twenty-six instructors from five clinical specialties participated in a scoring exercise at the College of Dentistry, University of Dammam, Saudi Arabia, in January 2015. Instructors blindly and independently scored 19-20 anonymous students' procedures, filling forms with rating scales as used in preclinical lab sessions. Their scores were compared to those of gold standard instructors in the same specialty. Percent agreement and having one letter grade difference from gold standard instructors were assessed, and associated factors were investigated after controlling for the effect of specialty/student. Agreement between instructors and their specialty gold standards occurred in 54.3% of decisions, with only 12.8% similar scores and 52.8% of scores representing underestimation in relation to gold standards. Instructor-related variables accounted for 40-50% of the variation in percent score as well as percent agreement and percent difference between instructors and gold standards. Educational background and teaching experience were significantly associated with less agreement among instructors and odds of one letter grade difference. Modest agreement existed among instructors in scoring various preclinical procedures, and there was a considerable percentage of one letter grade difference from gold standard instructors. Some factors associated with these outcomes are non-modifiable, but corrective measures are needed to increase agreement in scoring among instructors.