Evaluating success. Strategies and challenges for understanding IT implementation in a rural hospital

In 2004, a small rural hospital in California received a grant for the implementation of an integrated IT system. As part of the grant, the hospital worked with a university team to evaluate the implementation. The evaluation plan emphasized quantitative analysis of medication errors, patient safety and hospital finances. As the implementation progressed, it became clear that the quantitative methods would be marginally useful; qualitative methods gained greater importance. The evaluation team added more interviews and several staff surveys to the evaluation. The results of the surveys helped the hospital team understand how well staff was prepared for the technology and how they were responding to implementation. The best practices emphasized in this paper include: work with independent evaluators who can provide anonymity and lend objectivity; focus on surveys and interviews to understand the difficulties faced by staff and how they might affect outcomes; and collect key outcomes data in advance of beginning the implementation.