How to avoid over-fitting in multivariate calibration--the conventional validation approach and an alternative

Area of Science:

• Chemometrics
• Statistical Modeling
• Analytical Chemistry

Background:

• Over-fitting is a common problem in multivariate calibration, leading to models that do not generalize well.
• Conventional validation methods like cross-validation and independent test sets can be subjective and rely on 'soft' decision rules.

Purpose of the Study:

• To critically review over-fitting in multivariate calibration.
• To propose and evaluate a novel randomization test for assessing the statistical significance of model components.
• To compare this alternative approach with traditional validation techniques.

Main Methods:

• A randomization test was developed to assess the statistical significance of components in a multivariate model.
• This method was compared against cross-validation and independent test set validation.
• The methods were applied to a near-infrared spectral data set using partial least squares (PLS) regression.

Main Results:

• The randomization test provides a more objective assessment of model components compared to validation-based approaches.
• Unlike validation methods, the randomization test avoids the need for subjective 'soft' decision rules.
• The study demonstrated the effectiveness of the randomization test in the context of PLS regression for spectral data.

Conclusions:

• The proposed randomization test is a valuable and objective tool for multivariate calibration.
• It offers a statistically rigorous alternative to conventional validation techniques.
• This method enhances the reliability of chemometric models by providing objective significance assessments.