Design and analysis of classifier learning experiments in bioinformatics: survey and case studies

Area of Science:

• Bioinformatics
• Computational Biology
• Statistical Analysis

Background:

• Assessing algorithm performance is crucial in bioinformatics.
• Ensuring conclusions are statistically significant and not due to chance is vital.
• Proper experimental design and data analysis are key.

Purpose of the Study:

• To review performance measures, experiment design, and statistical tests in bioinformatics.
• To survey current practices in algorithm comparison within bioinformatics literature.
• To provide guidelines and case studies for robust statistical methodology.

Main Methods:

• Literature survey of 1,500 papers from three bioinformatics journals.
• Review of classification performance metrics and statistical testing basics.
• Analysis of four common bioinformatics scenarios with case studies.

Main Results:

• Basic experiment design principles like resampling are understood.
• Use of diverse performance metrics is common.
• Only 21% of surveyed papers employ statistical tests for algorithm comparison.

Conclusions:

• There is a significant gap in the application of statistical tests for algorithm comparison in bioinformatics.
• Adherence to rigorous statistical methodologies is needed for reliable bioinformatics research.
• Supplementary software aims to facilitate the adoption of discussed guidelines.