Empirical Performance of Cross-Validation With Oracle Methods in a Genomics Context

Area of Science:

• Statistical modeling
• Genomic data analysis
• Machine learning in bioinformatics

Background:

• Model selection methods like SCAD and Adaptive Lasso are commonly used with m-fold cross-validation.
• These methods perform well when regression functions are sparse with large signals.
• Genomic studies using Single Nucleotide Polymorphisms (SNP) present sparse regression functions with small signals.

Purpose of the Study:

• To investigate the performance of model selection methods in genomic regression with sparse signals.
• To evaluate the variability of variable selection in SCAD and Adaptive Lasso using cross-validation for SNP data.
• To question the reliability of single-run cross-validation for oracle methods in such contexts.

Main Methods:

• Empirical demonstration of variable selection performance.
• Utilizing m-fold cross-validation (m=10) with SCAD and Adaptive Lasso.
• Comparison with non-oracle methods like Lasso.

Main Results:

• Cross-validation exhibits considerable and surprising variation in selected variables for SCAD and Adaptive Lasso on SNP data.
• This variability persists even for non-oracle methods like Lasso.
• Small signal strengths in genomic data contribute to this instability.

Conclusions:

• A single run of m-fold cross-validation is questionable for oracle model selection methods in genomic studies.
• The reliability of variable selection is compromised by high variability in sparse, small-signal scenarios.
• Further research is needed to develop more robust cross-validation strategies for genomic data analysis.