Cross-study validation for the assessment of prediction algorithms

Area of Science:

• Biostatistics
• Machine Learning
• Bioinformatics

Background:

• High-dimensional prediction models are prevalent in statistical and machine-learning literature.
• Conventional cross-validation on exemplary datasets may not accurately reflect real-world prediction performance.
• Accurate prediction for independent samples in diverse settings is the primary goal in many applications.

Purpose of the Study:

• To introduce and implement a systematic 'cross-study validation' approach.
• To evaluate high-dimensional prediction models using independent datasets.
• To compare cross-study validation with conventional cross-validation.

Main Methods:

• Developed and implemented a cross-study validation framework.
• Utilized simulations and a collection of eight breast cancer gene-expression datasets.
• Computed C-index for pairwise training and validation datasets to assess distant metastasis-free survival (DMFS).

Main Results:

• Standard cross-validation yielded inflated discrimination accuracy compared to cross-study validation across all tested algorithms.
• The ranking of learning algorithms varied significantly between cross-validation and cross-study validation.
• Algorithms optimal under cross-validation may be suboptimal for independent validation.

Conclusions:

• Cross-study validation provides a more reliable assessment of prediction model performance in high-dimensional settings.
• The findings highlight the limitations of conventional cross-validation for real-world applicability.
• The proposed method is essential for selecting robust prediction algorithms for independent data.