Assessing performance of a randomized versus a non-randomized study design

Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.

The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality.

In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.

Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.