Norming a Written Communication Rubric in a Graduate Health Science Course

There is growing interest in the use of rubrics to assess written work. This study aimed to determine whether or not the norming of a written communication rubric improved scoring consistency among clinical faculty in a critical thinking course. The benefits of a formalized norming process are described.

Faculty-raters were trained to apply the rubric to a signature assignment while participating in calibration workshops. For each rubric criterion, faculty examined whether or not heightened congruence in scoring resulted from the training. Inter-rater reliability was determined after raters independently scored de-identified essays.

Pre-workshop intra-class correlations (ICCs) were acceptable (i.e., >0.7) for three of five rubric criteria. Post-workshop ICCs for only two criteria were acceptable: disciplinary conventions, and sources and evidence. Rater attrition and lag-time between calibration and post-workshop activities likely contributed to reduced consistency.

The rubric was useful for discriminating writing proficiency. Norming led to revision of the signature assignment, the rubric design, and a need for writing workshops. These changes will result in better student preparation for composing evidence-informed essays. Less-rigid approaches are worthy of future exploration.