The effect of rater severity on person ability measure: a Rasch model analysis

This paper presents a method for analyzing oral examinations with an extended, many-faceted Rasch model that calibrates medical specialty candidates, protocols, and raters. Significant variance was found among protocol difficulties and rater severities. When candidates' raw scores were compared with calibrated measures corrected for the bias caused by the particular protocols and raters encountered, variation between candidate scores and measures were observed. The data were found to fit the Rasch model well enough to be suitable for making measurement on oral examinations more objective as well as providing specific feedback to oral examination raters. In this example a medical oral examination was used; however, the techniques are applicable to any situation in which trained professionals rate candidate or patient performances. For occupational therapists, potential applications include evaluation of a student's fieldwork performance or observation of a patient's task performance.