Increasing inter-rater agreement on a family medicine clerkship oral examination--a pilot study

Although oral examinations are a traditional measure of student performance during clinical clerkships, concerns have been expressed about poor inter-rater reliability on these examinations. This study examines inter-rater agreement in a family medicine clerkship oral examination.

Our study analyzed oral examination scores awarded to a class of 98 junior medical students by three categories of examiner (the clerkship director, full-time faculty, and third-year residents). For each student, the difference among the three examiners' raw scores was compared using a Cronbach statistic. Mean scores for the year awarded by each of the three groups of examiners were compared using a correlation coefficient and paired t tests. For students with high inter-rater disagreement, examiner narrative comments and types of questions on which raters disagreed were analyzed.

Overall inter-rater agreement was high (Cronbach's alpha = .875). Paired t tests were nonsignificant between residents and faculty but were significant between residents and the clerkship director and between faculty and the clerkship director. In the small subset of students with little inter-rater agreement, no clear trends were identified to explain reasons for evaluator disagreement.

High levels of inter-rater agreement on a clerkship oral examination can occur when several measures to increase inter-rater agreement are used.