The effect of incomplete knowledge on the diagnoses of a computer consultant system

The knowledge bases (KBs) of diagnostic decision support systems are often incomplete, and gaps in the KB could potentially lead systems to reach diagnoses that are implausible to physicians. To investigate this possibility we studied Iliad (Version 2.01), a computer consultant system that generates differential diagnosis across the domain of internal medicine. Data from the history, physical examination, and laboratory findings of 50 grand-rounds cases were entered into Iliad by a computer consultant aware of the diagnosis but blinded to its presence or absence in Iliad's KB. Two experienced internists were asked to diagnose these cases before and after seeing the results of the computer consultation, and to assess the plausibility of the computer's diagnoses. Twenty-eight of the 50 cases (56.0%) were diseases contained in Iliad's KB. After seeing Iliad's diagnoses for cases in the KB, physicians assigned to their correct diagnoses a higher mean ranked position (1.5 versus 2.0, p less than 0.008) and a higher mean probability (84.0% versus 77.6%, p less than 0.008) compared with their pre-Iliad values, whereas for cases not in the KB, mean position and probability for correct diagnoses did not change. Physician diagnostic accuracy did not change after consultation on cases included or not included in the KB. After adjusting for case difficulty, mean plausibility of Iliad's diagnoses was judged significantly higher (on a seven-point scale) for cases in the KB than for cases not in the KB (4.2 versus 3.2, p less than 0.02).(ABSTRACT TRUNCATED AT 250 WORDS)