Policy and program analysis using administrative databases

Administrative policies and programs play an important and growing role as determinants of the use of medical care. Although some policies and programs may be harmful or ineffectual, randomized, controlled trials or prospective evaluations are rarely done for political or logistic reasons. Most evaluations are retrospective and often use administrative databases. Major problems with such evaluations include poor data quality, lack of concurrent controls, inability to ascertain important study outcomes, and incomplete data on case mix. This article uses published evaluations to illustrate these problems and suggests strategies that can minimize their impact. Such strategies include thorough assessment of data quality, interrupted time-series or policy gradient analysis, restriction of studies to those clinical outcomes that reliably result in medical care, and use of data on medical encounters as surrogates for determining case mix. However, even when these strategies are used, adequate evaluation of the effects of many policies and programs may continue to be impossible. Prospective evaluations need to be used more frequently to ensure that changes are held to the same standard used for other therapeutic interventions.