Bias correction for selecting the minimal-error classifier from many machine learning models.

Area of Science:

• Genomic research
• Machine learning
• Biostatistics

Background:

• Supervised machine learning is widely used in genomics for predictive classifier development.
• Cross-validation is a common technique for estimating error rates when independent test datasets are unavailable.
• A common practice of selecting the best performing model based on cross-validation can lead to selection bias, particularly in studies with moderate sample sizes.

Purpose of the Study:

• To address the selection bias in machine learning classifier error rate estimation.
• To propose and evaluate a novel bias correction method for cross-validation in genomic research.

Main Methods:

• Developed a bias correction method based on fitting learning curves using the inverse power law (IPL).
• Compared the proposed IPL method with existing techniques: nested cross-validation, weighted mean correction, and the Tibshirani-Tibshirani procedure.
• Evaluated methods using simulation datasets, moderate-sized real datasets, and large breast cancer datasets.

Main Results:

• The inverse power law (IPL) method demonstrated superior performance in bias correction compared to existing methods.
• IPL exhibited a smaller variance in error estimates.
• IPL offers the advantage of extrapolating error estimates for larger sample sizes, aiding decisions on sample recruitment.

Conclusions:

• The proposed IPL bias correction method is effective and offers practical advantages for genomic research.
• The 'MLbias' R package and source files are available for public use, promoting reproducibility and application.