Evaluating a comprehensive outpatient clinical information system: a case study and model for system evaluation

Decisions about information system implementation are often justified through a cost-benefit analysis. The ability to improve efficiency and outcomes while decreasing costs through information systems--by allowing for multiple and instant simultaneous access to information, through data monitoring and altering, through automation of protocols, and by collecting information for population-based health care as opposed to individual illness-care--are all potential benefits of a comprehensive clinical information system. Measuring the quantitative impact of these system improvements, however, is difficult. Doing a complete cost-benefit analysis of a comprehensive clinical information system is unrealistic due to the many assumptions necessary and the multiple confounding factors that are involved. In our Clinical Information Systems deployment in Kaiser Permanente, Northwest Region, we have elected not to do a detailed cost-benefit analysis. Instead, we have done an evaluation, based on success criteria, of a pilot implementation of a vendor-supplied system. This evaluation is based on clinician acceptance, system usage, technical factors, and quantitative effects on physician productivity. We also considered qualitative factors such as relationship with and responsiveness of the system vendor. We are moving ahead to regionalize this clinical information system based on such an evaluation of our pilot project. This paper outlines the approach that we have taken in evaluating our implementation of this system. It may provide some guidance for organizations on how to make a decision about whether or not to regionalize a clinical information system based on the evaluation of a pilot-site implementation.