Should treatment effects be estimated in pilot and feasibility studies?

Area of Science:

• Clinical Trials Methodology
• Biostatistics
• Evidence-Based Medicine

Background:

• Feasibility and pilot studies increasingly inform definitive randomized controlled trials (RCTs).
• These studies assess process measures (e.g., consent rates, fidelity) and provide parameter estimates for sample size calculations.
• However, using pilot data to estimate treatment effects for main trial planning is controversial.

Purpose of the Study:

• To critically evaluate the use of treatment effect estimates from pilot and feasibility studies in planning definitive randomized controlled trials (RCTs).
• To discuss the appropriate methods for determining sample size and progression criteria for main trials.

Main Methods:

• Methodological discussion of statistical approaches for using pilot study data.
• Critique of using between-group and within-group effect estimates from pilot studies for sample size and progression decisions.

Main Results:

• Estimating standard deviations from pilot data is empirical, but the minimum important difference (MID) for treatment effect should be based on clinical judgment.
• Pilot study treatment effect estimates are often imprecise and potentially biased, leading to unreliable sample size calculations and misguided progression decisions.
• Within-group effect estimates from pilot studies are insufficient for causal inferences regarding intervention efficacy.

Conclusions:

• Treatment effect estimates from pilot studies should not be used for main trial sample size calculations; the MID should be clinically determined.
• Progression decisions for main trials should not rely on imprecise pilot study treatment effects, which may also be biased.
• Robust planning for definitive RCTs requires careful consideration of clinical judgment over potentially misleading pilot data statistics.