An empirical comparison of expert-derived and data-derived classification trees

Classification trees provide an attractively transparent discrimination technique, and may be derived from both expert opinion and from data analysis. We consider a real and complex problem concerning the diagnosis of babies with suspected critical congenital heart disease into one of 27 classes. A full loss matrix for all possible misclassifications was obtained from clinical assessments. A tree derived from expert opinion was compared with those derived from analysis of 571 past cases, both for the full problem and for a subset of 6 diseases. Automatic methods for tree creation and pruning were found to have problems for rare diseases, and hand-pruning was carried out. Inclusion of costs led to much improved clinical performance, even for trees that had originally been constructed to minimize classification errors. The expert tree showed a specific building strategy that could not be reproduced automatically. The expert tree generally outperformed those derived from data, particularly in the ability to identify important composite features.