Treatment Effect Estimates From Pilot Trials Are Unreliable

The CONSORT guideline defines a pilot trial as a small-scale version of a desired future efficacy trial that is intended to answer the key questions of whether and how a larger study should be done. For example, a pilot trial might evaluate different approaches to data collection or outcome measurement. However, pilot trials are unreliable for assessing treatment efficacy due to the statistical phenomenon called sampling variability.

In this tutorial we use computer simulation to demonstrate the influence of sampling variability on efficacy estimates from pilot trials, illustrating why pilot trial designs should not be used to evaluate whether a treatment is promising or not.

We simulate a 2-arm parallel group trial (N=20 per group) with a survival outcome as an example. Simulations are done under two scenarios: 1) the treatment is efficacious at the level of a hypothetical minimum clinically important difference (hazard ratio [HR] = 0.75); and 2) the treatment is not efficacious (HR=1).

As expected, in both simulated scenarios the range of observed results is distributed around the true treatment effect, HR=0.75 or HR=1. Importantly, ∼20% of trials simulated under scenario 1 incorrectly suggest the treatment may be harmful (HR > 1). Under scenario 2, half of the simulated studies incorrectly suggest the treatment is beneficial.

Treatment effect estimates from pilot trials should not be used to make future development decisions regarding a novel therapy because of the high risk of misleading conclusions.