Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation.

Area of Science:

• Quantitative Structure-Activity Relationship (QSAR) modeling
• cheminformatics
• computational chemistry

Background:

• QSAR modeling requires robust model selection and validation due to lack of prior knowledge on optimal models.
• Accurate estimation of prediction errors (PE) is crucial but challenging, especially under model uncertainty, necessitating independent test sets.
• Double cross-validation (DCV), or nested cross-validation, efficiently generates test data and aids QSAR model selection, though its reliability under model uncertainty is debated.

Purpose of the Study:

• To systematically investigate the adequate parameterization of DCV for regression models combined with variable selection.
• To analyze the influence of the inner loop's cross-validation design and the outer loop's test set size on DCV performance.
• To address the controversy regarding DCV's reliability in estimating prediction errors under model uncertainty.

Main Methods:

• Analysis of simulated and real data using DCV to identify key factors influencing model quality.
• Application of bias-variance decomposition for simulated data to understand error sources.
• Systematic study of cross-validation design in the inner loop and test set size in the outer loop of DCV.

Main Results:

• Prediction errors of QSAR/QSPR regression models with variable selection are highly dependent on DCV parameterization.
• Inner loop parameters of DCV primarily impact model bias and variance.
• Outer loop parameters of DCV significantly influence the variability of the prediction error estimate.

Conclusions:

• DCV reliably and unbiasedly estimates prediction errors for regression models, even under model uncertainty.
• DCV provides a more realistic assessment of QSAR model quality compared to using a single test set.
• DCV should be preferred over a single test set for robust model evaluation in QSAR studies.