Cautionary Note on Using Cross-Validation for Molecular Classification.

Area of Science:

• Biomedical research
• Bioinformatics
• Molecular classification

Background:

• Reproducibility is a major concern in biomedical research.
• Published studies often face challenges in replication.
• Overly optimistic findings are linked to cross-validation in molecular classification.

Purpose of the Study:

• To investigate the impact of handling effects on cross-validation error estimation.
• To assess the role of data normalization in molecular classification studies.
• To identify methods for improving the reproducibility of molecular classification.

Main Methods:

• Utilized paired datasets with and without uniform sample handling.
• Simulated data using virtual rehybridization and various array-to-sample assignments.
• Estimated biological and handling effects from the datasets.

Main Results:

• Cross-validation underestimated error rates with confounding handling effects.
• Data normalization can exacerbate error underestimation.
• Balanced array assignment yielded unbiased error estimates.

Conclusions:

• Balanced array assignment enhances reproducibility in molecular classification.
• Caution is advised for routine use of data normalization and cross-validation.
• Addressing handling effects is crucial for accurate error estimation.