Multiple-rule bias in the comparison of classification rules

Area of Science:

• Bioinformatics
• Computational Biology
• Statistical Learning

Background:

• Bioinformatics community faces overoptimism in reported classification results.
• Two key issues contributing to overoptimism include reporting on favorable datasets and comparing multiple rules on a single dataset.

Purpose of the Study:

• To provide a probabilistic analysis of the 'multiple-rule bias' in classification.
• To quantify the bias in estimating the true error of the selected classification rule.
• To characterize the bias in estimating the comparative advantage of the chosen rule.

Main Methods:

• Probabilistic analysis of classification rule selection bias.
• Application of analysis to synthetic and real datasets.
• Utilizing various classification rules and error estimation methods.

Main Results:

• Quantification of the bias when selecting a classification rule with minimum estimated error.
• Characterization of bias in estimating the true comparative advantage of the selected rule.
• Demonstration of bias using both synthetic and real-world data.

Conclusions:

• The 'multiple-rule bias' is a significant factor contributing to overoptimism in classification performance reporting.
• Accurate estimation of classification rule performance requires accounting for the bias introduced by rule selection.
• The study provides a framework for more reliable evaluation of classification methods in bioinformatics.