Comparison of the ID3 algorithm versus discriminant analysis for performing feature selection

Having obtained disappointing results in a small medical data set despite the fact that our data seemed to be well suited for induction via ID3, we decided to compare the performance of ID3 to discriminant analysis. Performance was gauged by the percentage of correct classification in a second, independent data set. Examples were obtained from a cardiology project on the accuracy of auscultation. There were 107 examples in the first data set and 67 cases in the second. We found that ID3 and discriminant analysis performed equally poorly, with ID3 classifying only 60% of the second set correctly and discriminant analysis classifying 66% of the second set correctly. Also, the ID3 probability statistic for estimating the accuracy of ID3 for classifying further cases was markedly optimistic compared to our actual second data set results. Moreover, with an increase in sample size, ID3 seemed to break down, producing a large, complex decision tree of dubious generality, whereas discriminant analysis, with a larger sample size, used more independent variables but maintained its first set accuracy. These data suggest that there is a need for more sophisticated algorithms than ID3, even at the risk of giving up some computational efficiency.