Psychotherapist effects in meta-analyses: How accurate are treatment effects?

Psychotherapists are known to vary in their effectiveness with their clients, in randomized clinical trials as well as naturally occurring treatment settings. The fact that therapists matter has 2 effects in psychotherapy studies. First, if therapists are not randomly assigned to modalities (which is rare) this may bias the estimation of the treatment effects, as the modalities may have therapists of differing skill. In addition, if the data are analyzed at the client level (which is virtually always the case) then the standard errors for the effect sizes will be biased due to a violation of the assumption of independence. Thus, the conclusions of many meta-analyses may not reflect true estimates of treatment differences. We reexamined 20 treatment effects selected from 17 meta-analyses. We focused on meta-analyses that found statistically significant differences between treatments for a variety of disorders by correcting the treatment effects according to the variability in outcomes known to be associated with psychotherapists. The results demonstrated that after adjusting the results based on most small estimates of therapist effects, ∼80% of the reported treatment effects would still be statistically significant. However, at larger estimates, only 20% of the treatment effects would still be statistically significant after controlling for therapist effects. Although some meta-analyses were consistent in their estimates for treatment differences, the degree of certainty in the results was considerably reduced after considering therapist effects. Practice implications for understanding treatment effects, namely, therapist effects, in meta-analyses and original studies are provided.