An evaluation model for clinical information systems: clinical imaging systems

1. INTRODUCTION. Electronic patient records and other complex clinical information systems are difficult to evaluate. Integrated clinical imaging systems are a case in point. This paper suggests five criteria for a comprehensive evaluation of a complex clinical computer information system: 1) focus on a variety of technical, economic, and organizational concerns; 2) use multiple methods; 3) be longitudinal; 4) be formative as well as summative; and 5) be modifiable. 2. A MODEL COMPREHENSIVE EVALUATION PLAN. A model evaluation plan based on these criteria was developed for evaluating a clinical imaging system, but it provides a useful approach for a comprehensive evaluation of other complex clinical information systems. This plan consisted of three major phases. Phase 1 was to identify evaluation research questions and issues through literature synthesis and ethnographic interviews and observations. Evaluators interviewed key system users, potential users, developers, and other stakeholders. Also, evaluators observed the system being used. Phase 2 was to document system use through critical incident reporting and also by generating user profiles or by survey questionnaires. Phase 3 was to evaluate the system in context, so as to study it in different units and departments, in different wards and clinics, and at meetings, conferences, and rounds, etc. The proposed plan meets the above criteria. The plan employs multiple methods, intended to address a variety of evaluation questions and issues. It allows for longitudinal evaluation because phases should be repeated at periodic intervals so as to capture data at different times. The plan provides feedback for formative evaluation because each phase of the plan results in a product or output for individuals to use when making decisions concerning the system. These outputs can also serve as input to other phases and to a periodic refinement and validation of the evaluation plan. These modifications would also help ensure that the evaluation captures both anticipated and unanticipated concerns. In addition, the phases need not be performed sequentially. The order may be rearranged and some phases can be conducted concurrently. 3. AN EXAMPLE. A preliminary evaluation was conducted at the 700-bed Washington D.C. medical center run by the United States Department of Veterans Affairs [1, 2]. The evaluation used interviews and observations to document benefits of the imaging system--particularly clinical benefits--as identified by clinicians who used it. The clinical staff identified benefits they believed occurred in all the areas of an academic medical center: patient care, education, research, and administration. These benefits can be translated into cost savings, productivity improvements, and managerial and organizational benefits.