How a regression artifact makes ICFs/MR look ineffective

Based on a simple matched-control group quasi-experiment, Conroy (1996) concluded that small ICFs for persons with mental retardation have negative quality-of-life impacts. Our analysis of Conroy's design suggests, in contrast, that the reported effect is a pure regression artifact. The flaw in Conroy's design is selecting a control group on the basis of pretest matching. Although selecting a subsample of controls by matching on static characteristics such as age or gender can reduce the confounding influence of these variables, selection on the basis of pretest scores leads invariably to a large, spurious effect. The literature on this issue dates back a century, with warnings against pretest matching by Galton, Thorndike, McNemar, Stanley, Campbell, Cronbach, and Cook. We reviewed this historical literature and then used a Monte Carlo experiment to estimate the spurious effect that Conroy would observe from pretest matching alone. The magnitude of the artifact is as large as the quality-of-life reduction that Conroy attributed to the effects of living in an ICF. We discussed the methodological logic involved in matching and the broader policy issues raised by this evaluation.