Comparing groups in a before-after design: when t test and ANCOVA produce different results

Researchers often test people before and after some treatment and compare these scores with a control group. Sometimes it is not possible to allocate people into conditions randomly, which means the initial scores for the two groups may differ. There are two main approaches: t test on the gain scores and ANCOVA partialling out the initial scores. Lord (1967) showed that these can lead to different conclusions. This is an often-discussed paradox in psychology and education.

The reasons why these approaches can lead to different conclusions, the assumptions that each approach makes and how the approaches relate to group allocation, are discussed Methods. Three sets of simulations are reported that investigate the relationships among effect size, group allocation, measurement error and Lord's paradox.

Recommendations are given that stress careful examination of the research questions, sampling and allocation of participants and graphing the data. ANCOVA is appropriate when allocation is based on the initial scores, t test can be appropriate if allocation is associated non-causally with the initial scores, but often neither approach provides adequate results.