Shifting effects in randomised controlled trials of complex interventions: a new kind of performance bias?

Randomised controlled trials (RCTs) aim to provide unbiased estimates of treatment effects. However, the process of implementing trial procedures may have an impact on the performance of complex interventions that rely strongly on the intuition and confidence of therapists. We aimed to examine whether shifting effects over the recruitment period can be observed that might indicate such impact.

Three RCTs investigating music therapy vs. standard care were included. The intervention was performed by experienced therapists and based on established methods. We examined outcomes of participants graphically, analysed cumulative effects and tested for differences between first vs. later participants. We tested for potential confounding population shifts through multiple regression models.

Cumulative differences suggested trends over the recruitment period. Effect sizes tended to be less favourable among the first participants than later participants. In one study, effects even changed direction. Age, gender and baseline severity did not account for these shifting effects.

Some trials of complex interventions have shifting effects over the recruitment period that cannot be explained by therapist experience or shifting demographics. Replication and further research should aim to find out which interventions and trial designs are most vulnerable to this new kind of performance bias.