Can a reflective rubric be applied consistently with raters globally? A study across three countries

Reflection is a powerful tool for assisting students to develop the skills to make better informed decisions. As a pharmacy competency standard, reliable and fair assessment strategies are required to measure reflective skills and support students in developing their reflective capacity. The aim of this research was to explore whether we can extend the applicability of a previously tested rubric to a range of educational settings, to account for diversity of pharmacy educators and curricula internationally.

Four raters from three countries applied a reflective rubric to assess a sample (n = 43) of reflective accounts, representing 41% of a cohort of 105 second-year undergraduate pharmacy students. The interrater reliability (IRR) was measured utilizing the intra-class correlation coefficient (ICC), using a two-way random effects model with absolute agreement, to determine the level of agreement between the raters' absolute scores. Generalizability Theory analysis was used to estimate generalizability of raters and stages.

Results indicated agreement of raters for (i) each of the seven stages of reflection and (ii) overall score for the reflective account, with moderate to substantial agreement (ICC = 0.55-0.69, p < 0.001); and high agreement for all raters for the overall score (ICC = 0.96, p < 0.001), respectively. The G-Study estimated a relative error coefficient of 0.78.

This additional analysis further confirms the reliability and applicability of the rubric to a range of rater academic backgrounds.