Rater effects in evaluating therapist competencies using structured case reports: A mixed-methods study

Objective: This study applied a mixed-methods approach to investigate rater effects that might affect case report evaluation in China. Method: In the quantitative phase, we randomly assigned 210 mental health professionals to the experienced or novice ratee condition to rate the same structured case report on assessment and formulation, intervention strategy, relationship, and self-reflection. The qualitative phase was subsequently conducted to help make sense of the results. We interviewed twelve supervisors about how they would rate a case report and factors they thought would influence their evaluation. Results: Linear model with non-constant variances revealed that the manipulation of ratee experience did not significantly bias ratings. Raters whose self-identified theoretical orientation aligned with that of the report provided higher and less dispersed ratings for intervention strategy and therapeutic relationship competencies. As their years of experience increased, theoretically congruent raters tended to give lower and more dispersed ratings. Thematic analysis revealed highly variable cognitive processes, and identified rater theoretical expertise, evaluation context, rater subjectivity and dual relationships as important factors to consider. Conclusion: This study indicate that theoretically congruent raters would be desirable, but raters with more years of experience should be cautioned against giving stringent ratings for theoretically familiar cases.