Cognitive evaluation of decision making processes and assessment of information technology in medicine

This paper describes cognitive methods for analyzing medical decision making and evaluating medical information systems. The overall approach focuses on understanding the processes involved in the decision making and reasoning of health care workers, both with and without the use of information technologies. The issue of developing appropriate evaluation tools, for use in the design and analysis of medical information systems is considered to be of great importance. However, conventional methods are limited in their ability to identify and characterize the effects of information technology on the cognitive processes involved in decision making and reasoning. In this paper a range of methods are described involving video recording for collecting data on the use of information systems. The techniques described allow for the collection of an integrated data set consisting of transcripts of health care workers as they 'think aloud' in interacting with a medical system, along with complete video records of user-computer interaction. In addition, the methods can be extended to allow for the collection of process data from video recording of systems in actual clinical and emergency situations. The use of a variety of approaches, borrowing from research in cognitive science, is discussed. The development and application of these evaluation methods within the Canadian Centres of Excellence network HEALNet is subsequently described. Finally, implications for the development and evaluation of medical information systems are considered.