Striving for interrater reliability with a patient counseling assessment rubric: Lessons learned

Area of Science:

• Pharmacy Education
• Health Professions Education
• Assessment and Evaluation

Background:

• Effective patient counseling is a critical pharmacist skill for medication adherence and safety.
• Current assessment methods for patient counseling may lack consistency and objectivity.
• Developing standardized assessment tools is essential for pharmacist competency evaluation.

Purpose of the Study:

• To develop a novel patient counseling assessment rubric.
• To minimize interrater variability in student assessments.
• To provide consistent, actionable feedback for pharmacy students.

Main Methods:

• A patient counseling assessment rubric was developed.
• Study variables, including number of medications, evaluators, and videos, were systematically reduced.
• Interrater reliability was analyzed in two attempts.

Main Results:

• Neither the initial nor the refined attempt to analyze interrater reliability yielded statistically significant agreement.
• The study identified challenges related to the subjective nature of patient counseling assessments.
• Key lessons were learned regarding the evaluation of student pharmacist performance.

Conclusions:

• Achieving statistically significant interrater reliability for the patient counseling rubric was not successful.
• Recommendations include grader norming sessions, post-hoc analysis, and rubric simplification.
• The findings provide insights for institutions developing patient counseling curricula and assessment tools.