Exact test size and power of a Gaussian error linear model for an internal pilot study

Wittes and Brittain recommended using an 'internal pilot study' to adjust sample size. The approach involves five steps in testing a general linear hypothesis for a general linear univariate model, with Gaussian errors. First, specify the design, hypothesis, desired test size, power, a smallest 'clinically meaningful' effect, and a speculated error variance. Second, conduct a power analysis to choose provisionally a planned sample size. Third, collect a specified proportion of the planned sample as the internal pilot sample, and estimate the variance (but do not test the hypothesis). Fourth, update the power analysis with the variance estimate to adjust the total sample size. Fifth, finish the study and test the hypothesis with all data. We describe methods for computing exact test size and power under this scenario. Our analytic results agree with simulations of Wittes and Brittain. Furthermore, our exact results apply to any general linear univariate model with fixed predictors, which is much more general than the two-sample t-test considered by Wittes and Brittain. In addition, our results allow for examination of the impact on test size of internal pilot studies for more complicated designs in the framework of the general linear model. We examine the impact of (i) small samples, (ii) allowing the planned sample size to decrease, (iii) the choice of internal pilot sample size, and (iv) the maximum allowable size of the second sample. All affect test size, power and expected total sample size. We present a number of examples including one that uses an internal pilot study in a three-group analysis of variance.