Implementation Variation in Natural Experiments of State Health Policy Initiatives

An increasing number of federal initiatives allow states flexibility in selecting the strategies used to achieve initiative-specific goals. Variation in the foci and intensity of implementation may explain why federal policy initiatives succeed in some states and fail in others. The CMS State Innovation Models (SIM) initiative is a complex policy intervention implemented with substantial variation across states and may have variable impacts. This paper presents a method to characterize and account for that variation in states' implementation foci and intensity in natural policy experiments.

A combination of quantitative and qualitative measures of SIM implementation was used to characterize the foci of payment and delivery system reforms across states.

A modified Delphi expert panel process was used to prioritize the features of SIM implementation that would differentiate grantee states with respect to improved health outcomes. Three researchers then reviewed summaries of published evaluations and reports to characterize and score states on each implementation feature. Expert panelists guided the researchers on developing the criteria and weights applied to the focus areas when calculating SIM implementation intensity scores for states.

Over 3 years of an expert panel process, 4 dimensions of SIM implementation that would most affect health outcomes were prioritized: 1) extent and breadth of stakeholder engagement, (2) extent that SIM implementation was focused on improving behavioral health, (3) amount of SIM funding per capita, and (4) breadth and depth of value-based payment reforms. Scoring states based on the prioritized factors resulted in composite scores that differentiated states into 3 categories: high, moderate, and low implementation intensity.

We developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across grantee states. Our method for characterizing state implementation variation may be useful for natural policy experiments examining the variable impact of policy initiatives.