Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation

Area of Science:

• Bioinformatics
• Computational Biology
• Statistical Genetics

Background:

• Publicly available biological data facilitates large-scale analyses by combining multiple datasets.
• Technical variations (batch effects) and sample composition differences between datasets can compromise generalizable conclusions.

Purpose of the Study:

• Investigate the impact of batch effects and sample composition on classification performance estimates.
• Focus on estimation bias in cross-validation for classifier performance.
• Differentiate from prior studies by emphasizing estimation bias over predictive performance.

Main Methods:

• Utilize simulated gene expression data based on real data for realistic intensity distributions.
• Introduce batch effects and differential expression between simulated control and treated groups.
• Employ nested cross-validation for classifier training, feature selection (Wilcoxon test, lasso), and performance estimation.

Main Results:

• Batch effects and differing sample compositions introduce significant bias in cross-validation estimates of classifier performance.
• The degree of confounding between group and batch effects influences the magnitude of the estimation bias.
• Classifier performance estimates are more accurate when accounting for batch effects and sample composition.

Conclusions:

• Cross-validation estimates of classifier performance can be unreliable when batch effects are present.
• Careful consideration of batch effects and sample composition is crucial for accurate performance evaluation.
• The findings underscore the importance of robust methods for data integration in multi-dataset biological studies.