Reducing confounding in natural experiments: evaluating a multilevel matching approach

The current study examined the utility of rigorous multilevel matching procedures in a large national experiment examining the effectiveness of the Stepping Up (SU) Initiative for increasing evidence-based behavioral health services to justice-involved individuals in and out of jail.

Data for the current study were derived from surveys of administrators in 133 SU and 133 non-SU matched comparison counties. We used hierarchical linear modeling to test whether our pairing method successfully removed confounding variables from influencing baseline values of outcomes.

The multilevel case-controlled matching procedure reduced number of confounding factors (from 9 to 4 over 3 tested models). It improved predictive models by accounting for differences in baseline county characteristics. Analyses identified additional variables (e.g., recent funding changes) to consider in future matching procedures. Matched models yielded fewer significant predictors, reflecting successful control of baseline differences and clearer isolation of SU-related effects.

Multilevel case-controlled matching procedures can reduce bias due to confounding variables in baseline values of outcomes. Multilevel case-control matching is recommended in natural experimental designs in criminal justice and health research.