Testing Three Alternate Methods to Direct Observation in Measuring Use of Discrete Youth Cognitive Behavioral...

Area of Science:

• Psychiatry and Behavioral Health
• Implementation Science
• Clinical Psychology

Background:

• Cognitive behavioral therapy (CBT) is an evidence-based treatment for youth psychiatric conditions.
• Accurate measurement of CBT technique delivery is crucial for quality assessment but remains challenging.
• Existing low-cost methods like clinician self-report have not been adequately studied for measuring specific CBT techniques.

Purpose of the Study:

• To evaluate the concordance of self-report, chart-stimulated recall (CSR), and behavioral rehearsal (BR) with direct observation (DO) for measuring individual CBT techniques.
• To determine if concordance differs between cognitive and behavioral CBT techniques across measurement methods.

Main Methods:

• Clinicians (n=126) were randomized to self-report, CSR, or BR conditions.
• The study analyzed the agreement between these methods and direct observation (DO) for discrete CBT techniques.
• Mixed-effects regression models were used to compare scores from each method against DO.

Main Results:

• Behavioral rehearsal (BR) scores showed no significant difference from direct observation (DO) for any CBT technique.
• Chart-stimulated recall (CSR) scores did not differ from DO for nine out of twelve techniques.
• Self-report scores significantly differed from DO for most techniques, with higher concordance for behavioral techniques compared to cognitive ones.

Conclusions:

• Both BR and CSR offer viable, less resource-intensive alternatives to direct observation for measuring most CBT techniques.
• These findings support the use of multiple measurement methods in implementation research and clinical practice.
• Chart-stimulated recall and behavioral rehearsal are more accurate than self-report for assessing CBT technique fidelity.