Assessing Therapist Competence: Development of a Performance-Based Measure and Its Comparison With a Web-Based...

Area of Science:

• Psychological treatment training and assessment
• Therapist competence evaluation
• Digital health in mental healthcare

Background:

• Assessing therapist competence requires evaluating both knowledge and practical skills.
• Current skill assessment methods are often labor-intensive and not scalable for training outcomes.
• There is a need for rigorous yet efficient methods to measure therapist competence.

Purpose of the Study:

• To develop and evaluate a role-play-based measure for assessing therapist skill.
• To compare the role-play measure's performance with a scalable web-based knowledge measure.
• To investigate the web-based measure's ability to predict competence assessed via role-play.

Main Methods:

• Developed and piloted role-play scenarios and a rating scheme using enhanced cognitive behavioral therapy (CBT-E) for eating disorders.
• Evaluated performance of 93 therapists with varying backgrounds and training levels in CBT-E.
• Administered a web-based applied knowledge measure and analyzed its predictive validity for role-play competence.

Main Results:

• The role-play measure demonstrated good-to-excellent inter-rater reliability and sensitivity to training changes.
• Therapists' self-rated performance in role-plays generally reflected their usual clinical practice.
• The web-based measure efficiently predicted therapist competence, showing 77% positive predictive value and 78% specificity.

Conclusions:

• The role-play measure is feasible and acceptable for assessing therapist performance.
• The web-based measure offers a scalable and cost-effective alternative for evaluating therapist competence in certain contexts.
• This study provides a viable approach for assessing training outcomes in psychological therapies.