Evaluation of a new method for cardiovascular reasoning

Evaluate the accuracy of the detailed diagnostic reasoning of the Heart Failure Program incorporating a new mechanism to handle temporal relationships and severity constraints.

Tools were developed to summarize diagnoses and automatically generate evaluation forms. Five expert cardiologists were asked to review the reasoning of the program, with two analyzing each case. Cases were gathered retrospectively for diversity and difficulty and 26 randomly selected cases were evaluated. The underlying issues were identified and classified.

Both reviewers rated the first diagnosis correct in 25% of the cases and at least one rated it wrong in 10%. Analyzing the detailed reasoning, 137 issues were raised, about 5.3 per case. Of these, 53% were possible concerns raised by one reviewer. Of the 5.3 issues per case, 2.5 were attributable to controversies, misunderstandings, or mistakes; 1 was due to the overly simplistic representation of the summaries; and 1.8 were issues related to the program.

Overall, the program is capable of providing high-quality detailed diagnostic hypotheses for complex cardiovascular cases. The results highlight several issues: 1) the difficulty of effectively summarizing hypotheses, 2) the nature of a physician's causal explanation, and 3) some problems in evaluating detailed diagnostic reasoning. The mistakes the program made imply that some additional refinement is needed but that the reasoning mechanisms developed can support the appropriate reasoning. The appropriate next step is a prospective evaluation addressing the program's usefulness.