Evaluation stages and design steps for knowledge-based systems in medicine

After the early experiments in artificial intelligence a methodology is emerging around advanced systems for the management of medical knowledge. The stress is moving away from the implementation of prototypes to the evaluation. It is possible to adapt and to apply this to field evaluation techniques already developed in similar contexts of knowledge management (books, drugs, epidemiology, consultants, etc.). The time is ready for a further step: to envisage a methodology for the design of real systems that cope with the 'knowledge environment' of the user. Every stage of the evaluation process is re-examined here, and considered as a framework to define goals and criteria about a step of design: (1) the impact of the system on the progress of health care provision (priorities, cost-benefit analysis, share of tasks among different media); (2) effectiveness in the end-user's environment and long-term effects on his behaviour (changes in people's role and responsibilities, improvements in the quality of data, acceptance of the system); (3) the intrinsic efficiency of the system apart from the operational context (correctness of the knowledge base, appropriateness of the reasoning). The need to differentiate the test sample into three classes (obvious, typical, atypical) is emphasized, discussing the influence on both evaluation and design. In particular the difficulty of having 'gold standards' on atypical cases, due to the disagreement among the experts, leads to the definition of two alternative attitudes: the 'standardization mode' and the 'brain-storming mode'.