Grading written essays: a reliability study

The purpose of this study was to examine the interrater reliability of grades obtained by physical therapy and occupational therapy tutors in rating their students' term papers. This study was carried out in two phases. In phase 1, four student essay papers (two physical therapy students' papers and two occupational therapy students' papers) with grades that had been assigned from the previous year's course were randomly selected from a bank of papers. These papers were independently rated by three course planners (who were responsible for planning, coordinating, and tutoring in the course), and agreement as to the assignment of grades for each paper was established. In phase 2, the same students' essays were rated independently by eight course tutors. To test for differences among students' written essay papers and for differences among the raters in the subcategories of discipline (physical therapy versus occupational therapy) and level of expertise (novice versus experienced) in grading essays, a three-way analysis of variance was performed. An intraclass correlation coefficient (ICC) was calculated for interrater reliability. Although there were no statistically significant differences among the tutors with regard to their discipline and expertise, the reliability analysis produced an ICC of .79. Strategies to enhance the reliability of grading essays are discussed.