The consequences of ignoring therapist effects in trials with longitudinal data: A simulation study

Psychotherapy trials frequently generate multilevel longitudinal data with 3 levels. This type of hierarchy exists in all trials in which therapists deliver the treatment and patients are repeatedly measured. Unfortunately, researchers often ignore the possibility that therapists could differ in their performance and instead assume there is no difference between therapists in their average impact on patients' rate of change. In this article, we focus on scenarios in which therapists are fully and partially nested within treatments and investigate the consequences of ignoring even small therapist effects in longitudinal data.

We first derived the factors leading to increased Type I errors for the Time × Treatment effect in a balanced study. Scenarios with an unbalanced allocation of patients to therapists and studies with missing data were then investigated in a comprehensive simulation study, in which the correct 3-level linear mixed-effects model, which modeled therapist effects using a random slope at the therapist level, was compared with a misspecified 2-level model.

Type I errors were strongly influenced by several interacting factors. Estimates of the therapist-level random slope suffer from bias when there are very few therapists per treatment.

Researchers should account for therapist effects in the rate of change in longitudinal studies. To facilitate this, we developed an open source R package powerlmm, which makes it easy to investigate model misspecification and conduct power analysis for these designs. (PsycINFO Database Record