A performance evaluation of the expert system 'Jaundice' in comparison with that of three hepatologists

The diagnostic performance of an Expert System (Jaundice) designed to discriminate between different causes of jaundice was evaluated in a test sample of 200 consecutive in-patients with serum bilirubin greater than or equal to 51 mumol/l. The average probability assigned to true diagnosis, the non-error rate and the overall accuracy were, respectively, 55%, 77% and 70%. The Expert System's discriminatory ability in probabilistic prediction, assessed by a method based on continuous functions of the diagnostic probabilities (Brier score) was good. We also compared the ability of our Expert System to that of three experienced hepatologists, who were required to give a diagnosis in 20 cases following the same protocol used by computer (i.e., by asking only clinical and laboratory items). Both the hepatologists and Jaundice achieved a correct diagnosis in 70% of 20 cases, but the Expert System asked a significantly higher average number of questions during each consultation. Analysis of the reasoning pathway made by an external referee showed a high agreement between the diagnostic strategies of the Expert System and the physicians. We conclude that Jaundice can be a useful tool to support a physician with insufficient clinical experience in this field to generate correct diagnostic hypotheses.