Improving interrater agreement in evaluation in dentistry by the use of comparison stimuli

Research shows that use of rating scales with precisely defined scale points improves interrater agreement. Instead of words, scale points may be defined by products for which ratings have been established by consensual agreement. These products serve as standards for raters. In two studies, raters assessed student products using conventional rating scales and identical scales together with comparison stimuli (photographic exemplars) of consensually established values of 3 (adequate) for each criterion. Results of the first study (but not the second) clearly confirmed the contribution of exemplars to interrater agreement. Findings suggested that single exemplars significantly enhance agreement for criteria on which the ideal is an end point on a continuum. Multiple exemplars may be necessary for other criteria. Comparison stimuli offered promise for improving interrater agreement, but further work is necessary to adapt their use for complex criteria that frequently define the quality of dental preparations.