An approach to evaluating the accuracy of DXplain

DXplain is a computer-based decision support system which generates a differential diagnosis (ddx) from a given list of clinical manifestations (Barnett et al., J. Am. Med. Assoc. 258 (1987) 67-74). An approach was developed to evaluate the accuracy of the ddx's produced by DXplain. The first step involves the collection of 65 benchmark cases drawn from a variety of sources and authors. Despite their diverse origins, the cases share in common that they are all clinical cases upon which a consulting physician might be asked to produce a differential. This helps to ensure that the evaluation of the system will be done in an environment similar to that in which the system is actually used. In the second step, all cases are reviewed by five board-certified physicians (experts) as well as DXplain. For each case, the evaluators (experts and DXplain) produce a rank-ordered ddx list along with an indication of how strongly each disease was felt to be supported by the case findings. A scoring technique was devised which rewards concordance with the gold standard: a consensus of the evaluators' ddx lists. Each evaluator receives a score which is proportional to the degree of agreement achieved with the consensus on the ddx submitted. Preliminary results on a trial evaluation of 46 cases indicate that DXplain, on average, did well in agreeing with the consensus. Agreement was achieved both in regard to the specific diagnoses listed in the ddx and the degree to which the diseases were felt to be supported by the case findings. A discussion of some important issues in the evaluation of knowledge-based systems is undertaken.