Cross-study validation and guidance for resampling approaches for partial least squares discriminant analysis in...

Area of Science:

• Spectroscopic Machine Learning (SML)
• Chemometrics
• Statistical Learning

Background:

• Resampling strategies are crucial for model selection and performance evaluation in SML.
• Their influence is often underestimated compared to model choice.

Purpose of the Study:

• To systematically evaluate how common resampling methods affect model selection, predictive performance, and estimation bias.
• To compare resampling strategies within a fixed partial least squares-discriminant analysis (PLS-DA) framework.

Main Methods:

• Evaluation across twelve independent spectroscopic datasets.
• Model tuning using the one-standard-error (1SE) rule and highest Macro F1 (HMF1) selection.
• Performance assessment using repeated external validation and Bayesian Bradley-Terry analysis.

Main Results:

• Bootstrap and jackknife methods best preserved predictive performance.
• LOOCV, Venetian blinds, and some K-fold approaches balanced performance stability and model parsimony.
• Five-fold K-fold cross-validation yielded the least biased Macro F1 estimates, outperforming bootstrap and LOOCV.

Conclusions:

• Resampling strategy is a primary determinant of model selection and performance estimation accuracy in SML.
• K-fold cross-validation provides more reliable performance estimates than bootstrap or LOOCV.
• This study offers a framework for selecting optimal resampling methods to enhance SML model reliability and interpretability.