Implications of therapist effects for the design and analysis of comparative studies of psychotherapies

Technical reasons are presented as to why therapist should be included as a random design factor in the nested analysis of (co)variance (AN[C]OVA) design commonly used in psychotherapy research. Incorrect specification of the ANOVA design can, under some circumstances, result in incorrect estimation of the error term, overly liberal F ratios, and an unacceptably high risk of Type I errors. Review of studies indicates that the great majority of investigators continue to ignore this issue. Computer simulation studies revealed that considerable bias can be introduced by not specifying therapist as a random term. Finally, a reanalysis is presented of data from 10 psychotherapy outcome studies that indicated that therapist effects vary considerably and at times can be large. More recent studies that implement better quality controls appear to demonstrate less variance due to therapist. The implications of these results for the design of future studies are discussed.