Evaluating implementation fidelity in health information technology interventions

Health information technology evaluators need to distinguish between intervention efficacy as assessed in the ideal circumstances of clinical trials and intervention effectiveness as assessed in the real world circumstances of actual practice. Because current evaluation study designs do not routinely allow for this distinction, we have developed a framework for evaluation of implementation fidelity that considers health information technologies as complex interventions and makes use of common intervention components as defined in the Oxford Implementation Index. We also propose statistical methods for the evaluation of interventions at the system and component level using the Rubin Causal Model. We then describe how to apply this framework to evaluate an ongoing clinical trial of three health information technology interventions currently implemented in a 17,000 patient community-based health network caring for Medicaid beneficiaries in Durham County, North Carolina.