Evaluation of a medical diagnosis system using simulator test scenarios

This paper describes an informal but systematic method for how to test and verify a knowledge-based system in a large open-ended medical target domain. The system used is Guardian, an intelligent system for monitoring and diagnosis of post-cardiac surgery patients in an intensive-care unit. The knowledge base is tested and verified by running the system on a series of realistic test scenarios, both with an embedded simulator and with an external simulation system. The same scenarios are presented to human test subjects, making it possible to compare and analyze the performance of the knowledge-based system with that of human physicians. The use of simulators instead of clinical data also means that it is possible to test crucial scenarios which occur seldom in medical practice. Our results show that a system like Guardian might indeed be useful in medical care.