Sample-size dependence of validation parameters in linear regression models and in QSAR.

Area of Science:

• Chemometrics
• Statistical Modeling
• Data Science

Background:

• Accurate statistical validation is crucial for reliable multivariate linear models.
• Internal validation parameters (e.g., R-squared) can be misleading with small sample sizes.
• Understanding the impact of sample size on validation metrics is essential.

Purpose of the Study:

• To investigate the dependence of statistical validation parameters on sample size in multivariate linear curve fitting.
• To compare the performance of internal, cross-validation, and external validation metrics.
• To provide guidance on selecting appropriate validation strategies based on sample size.

Main Methods:

• Analysis of R-squared and related internal parameters.
• Application of cross-validation techniques (leave-one-out, leave-many-out).
• Calculation of y- and x-randomized validation parameters and Roy-Ojha metrics.
• Rank correlation analysis between different validation parameters.

Main Results:

• R-squared and internal parameters overestimate model fit with small sample sizes.
• Cross-validation metrics (leave-one-out, leave-many-out) provide accurate trends after adjusting degrees of freedom.
• External parameters show correct trends but vary in sensitivity; Roy-Ojha metrics aid model classification.
• Internal robustness checks become redundant above a certain sample size.

Conclusions:

• Internal validation parameters are unreliable for small sample sizes.
• Cross-validation and external validation are necessary for robust model assessment, especially with limited data.
• All three validation aspects (goodness-of-fit, robustness, external validation) are vital for small sample sizes, but internal robustness loses informativeness with larger samples.