Evaluating and validating very large knowledge-based systems

Most knowledge-based systems for use in medicine have been developed in response to specific problems such as the diagnosis of abdominal or chest pain in an accident and emergency department, or the diagnosis and treatment of meningitis. There is a role for a general decision support system capable of answering queries about any aspect of medicine, particularly in primary care. However, evaluating such a knowledge base requires more elaborate methodology than a simple iterative test and refine cycle. At the design stage an adequate knowledge base structure is required to allow focused modification of the knowledge base when errors are discovered. During the prolonged evaluation cycle the partially formed knowledge base must be tested with such techniques as validation checks for consistency and completeness and examination of characteristics of problem-solving procedures. Finally a variety of criteria that represent the performance, robustness, flexibility, predictability, validity, coverage, relevance and congruity of the knowledge base are needed for a full description of the system's worth. Two case studies from the Oxford System of Medicine project are provided as examples of this philosophy: validating specific medical facts and comparing two methods for aggregating reasoning for and against a decision option.