Interrater concordance for faculty grading of student performances in a problem-based learning course

To determine whether it is possible for faculty to arrive at consistent, non-idiosyncratic grades in a problem-based learning (PBL) course.

Integrated Case Studies and Medical Decision Making (ICS) is the final course of the second year at the University of Pittsburgh School of Medicine. In ICS, 16 groups of nine students work in a PBL format over seven weeks. Each group is led by three faculty facilitators who, at the end of the course, independently give each student ratings for overall performance in the course and for each of seven performance categories. In 1993-94 and 1994-95, concordance in grades among the facilitators was determined by computing the intraclass correlation coefficients [ICC (3,1)] for the overall scores, the seven performance category scores, and all eight scores in aggregate. An ICC (3,1) of > or = .1 was considered indicative of statistically significant interrater concordance. An ICC (3,1) of > or = .7 was considered indicative of concordance of practical significance.

Because the facilitators occasionally did not rate every student in every performance category, complete information was not available for all 32 groups. Statistically significant concordance was achieved in the aggregate scores in 100% of 23 groups, and in the overall scores in 90% of 18 groups. In six of the seven performance categories, concordance was achieved in at least 75% of the groups (n = 16-20). Practically significant concordance was achieved in the aggregate scores in 83% of 23 groups.

The study results show that, given specific criteria by which to judge students' performances, it is possible to arrive at consistent, non-idiosyncratic grades for students in PBL courses.