On the optimistic performance evaluation of newly introduced bioinformatic methods

Area of Science:

• Bioinformatics
• Computational Biology
• Epigenetics

Background:

• Research articles frequently claim novel data analysis methods offer superior performance.
• The veracity of these claims is often questionable due to inherent biases in evaluation.
• Optimistic bias can arise from dataset selection, choice of competing methods, and selective reporting.

Purpose of the Study:

• To discuss and illustrate the consequences of optimistic bias in evaluating new data analysis methods.
• To quantitatively investigate optimistic bias using a specific example in epigenetic analysis.
• To highlight the impact of bias on the reported performance of data analysis techniques.

Main Methods:

• Illustrative examples and quantitative investigation of optimistic bias.
• Focus on normalization methods for Illumina HumanMethylation450K BeadChip microarray data.
• Analysis of biases stemming from dataset selection, competing method selection, and bug fixing.

Main Results:

• Demonstration of how optimistic bias can lead to exaggerated performance claims for new methods.
• Quantitative evidence of bias in the context of epigenetic data normalization.
• Identification of specific sources of bias, including selective reporting and method variant reporting.

Conclusions:

• Optimistic bias is a significant issue in the evaluation of data analysis methods.
• The findings underscore the need for rigorous and unbiased evaluation protocols.
• Improved transparency and standardized benchmarking are crucial for reliable method assessment in bioinformatics.