[A principal components selection method based on the modified randomization test for avoiding over-fit and under-fit...

Area of Science:

• Chemometrics
• Spectroscopy
• Data Analysis

Context:

• Quantitative calibration models using principal components analysis (PCA) are prone to over-fitting or under-fitting.
• Selecting the optimal number of principal components is crucial for accurate spectral calibration.
• Existing methods like cross-validation can be suboptimal, especially with complex datasets.

Purpose:

• To propose a novel principal component selection method based on a modified randomization test.
• To evaluate the proposed method's effectiveness in avoiding over-fitting and under-fitting in spectra calibration.
• To compare the new method with traditional cross-validation.

Summary:

• A modified randomization test is introduced for principal component selection in spectra calibration, addressing over-fitting and under-fitting.
• The method considers all training samples, unlike cross-validation, leading to improved prediction performance.
• Experiments with near-infrared spectra demonstrated the method's adaptability to complex samples and its ease of implementation with a visualized process.

Impact:

• The proposed method enhances the prediction performance of quantitative calibration models in spectral analysis.
• It offers an adaptable and robust approach for selecting principal components, even with complex and limited sample data.
• The visualized and interactive selection process facilitates practical application in spectral data analysis.