Striving for interrater reliability with a patient counseling assessment rubric: Lessons learned

Providing effective patient counseling is an essential pharmacist skill to ensure patients understand how to take medications, prevent medication-related errors, and meet requirements of federal law. This study sought to develop a new patient counseling assessment rubric to minimize interrater variability, deliver a consistent summative competency assessment, and provide students with formative, actionable feedback.

A first attempt to achieve statistically significant interrater reliability was not successful due to incorporation of too many variables into study design and the subjective nature of patient counseling. After reducing study variables (number of different medications, number of evaluators, and number of videos) and consulting a statistician, a second attempt was made to analyze interrater reliability for the rubric. However, even with variables minimized, this attempt did not lead to statistically significant agreement.

The faculty team identified four recommendations (Omnibus Budget Reconciliation Act of 1990): conduct a norming session for graders prior to the assessment (Rantucci, 2006), conduct a post-hoc analysis after grading to reduce interrater variability and increase consistency (Taitel et al., 2012), simplify the rubric to reduce subjectivity and clarify the intent of rubric elements, and (Saranagam et al., 2013) rubrics can be utilized differently in separate courses to target specific learning objectives.

Although the goal of creating a rubric with statistically significant interrater reliability was not achieved, we did learn important lessons about evaluating student pharmacist performance with less subjectivity and more consistency. The authors hope the results and lessons learned will be valuable to our colleagues at other institutions as patient counseling content and rubrics are developed.