Performance evaluation of medical expert systems using ROC curves

Medical diagnostic systems are increasingly used to support clinicians in complex decision-making processes. However, the performance of these systems often remains unclear, especially in specialized areas like pancreatic disease diagnosis. Prior research has shown that expert systems can assist in diagnostic reasoning, but their accuracy has not been fully evaluated. This uncertainty drives the need for objective performance metrics that can be applied consistently. Traditional diagnostic accuracy measures may not capture the full scope of an expert system's capabilities. No prior work had resolved how to evaluate the impact of data completeness on system performance. This gap motivated the use of receiver operating characteristic (ROC) curves to assess diagnostic systems comprehensively. Such an approach allows for a nuanced understanding of how data availability affects diagnostic accuracy.

The aim of this study was to evaluate the diagnostic accuracy of the CADIAG-2/PANCREAS expert system using a structured approach. The researchers sought to determine how well the system performs when given limited versus complete patient data. They also wanted to assess whether the gold standard diagnosis appears in the system's generated hypotheses. A key motivation was to explore how the diagnostic ranking changes with the inclusion of additional diagnostic tests. The study aimed to provide a framework for comparing medical expert systems using ROC curves. By varying an internal threshold, the researchers could observe how diagnostic hypotheses are generated. This approach allows for a more detailed understanding of the system's diagnostic behavior. The ultimate goal was to determine whether CADIAG-2 can be effectively used at early stages of the diagnostic process.

The study evaluated the diagnostic performance of CADIAG-2/PANCREAS using 47 clinical cases from a university hospital. Each case was assessed twice: once with limited data and once with full diagnostic information. The gold standard was defined as histologically or clinically confirmed diagnoses. The first evaluation focused on patient history, physical examination, and basic laboratory tests. The second evaluation included additional diagnostic data such as imaging and biopsy results. The system's diagnostic hypotheses were ranked, and the gold standard's position was recorded. Receiver operating characteristic (ROC) curves were generated by adjusting an internal threshold. This allowed the researchers to analyze how varying diagnostic criteria affected system performance.

The gold standard diagnosis was typically included in the initial list of hypotheses generated by CADIAG-2. Using only basic data, the system's top hypothesis matched the gold standard in most cases. When complete data were available, the gold standard was ranked first in the majority of cases. Exceptions occurred in patients with chronic diseases where findings were nonspecific. ROC curves demonstrated that adjusting internal thresholds can optimize the system's performance. The system's ranking improved significantly with the inclusion of additional diagnostic tests. This suggests that CADIAG-2 can be effective even at early diagnostic stages. The results also indicated that ROC curves provide a useful framework for comparing diagnostic systems.

The authors suggest that CADIAG-2/PANCREAS can be used effectively at early stages of the diagnostic process. The system's ability to include the gold standard in its initial hypotheses is a notable finding. They propose that ROC curves are a valuable tool for optimizing diagnostic systems. The study shows that complete patient data significantly improves diagnostic accuracy. The researchers suggest that ROC curves can also be used to compare different diagnostic systems. They emphasize that the system's performance is sensitive to the completeness of the input data. This implies that the system's utility is closely tied to the availability of comprehensive diagnostic information. The findings support the use of ROC curves as a standard for evaluating medical expert systems.