The evaluation of artificial intelligence systems in medicine

This paper discusses the underlying issues in the evaluation of computer systems which apply artificial intelligence in medicine. Three different levels of evaluation are described: the subjective evaluation of the research contribution of a developmental prototype, the validation of a system's knowledge and performance, and the evaluation of the clinical efficacy of an operational system. The paper outlines a number of evaluation issues at each level, and discusses how previous artificial intelligence in medicine evaluations fit into this framework.