An empirical assessment of validation practices for molecular classifiers

Area of Science:

• Biostatistics
• Bioinformatics
• Genomics and Proteomics

Background:

• Molecular classifiers are susceptible to overfitting with noisy genomic and proteomic data.
• Cross-validation is commonly used for accuracy estimation but can introduce bias.
• External validation is essential for assessing generalizability and bypassing bias.

Purpose of the Study:

• To evaluate the practices and impact of external validation for molecular classifiers.
• To compare performance estimates from internal cross-validation versus external validation.
• To assess the power of studies to detect performance discrepancies between validation methods.

Main Methods:

• Reviewed 35 studies reporting external validation of molecular classifiers.
• Extracted study design and methodological features.
• Compared internal cross-validation and external validation performance in 28 studies.
• Analyzed study power to detect performance decreases and diagnostic odds ratios.

Main Results:

• Most studies used cross-validation methods likely to overestimate classifier performance.
• Median sensitivity and specificity were higher in cross-validation (94%, 98%) than external validation (88%, 81%).
• Studies were underpowered to detect performance drops; median power was 36% for sensitivity and 29% for specificity.

Conclusions:

• Cross-validation practices in molecular classifier studies are often biased.
• Routine external validation is necessary for genuine progress in the field.
• Future studies require larger sample sizes for robust external validation of molecular classifiers.