Outlier Removal with Weight Penalization and Aggregation: A Robust Variable Selection Method for Enhancing...

Area of Science:

• Analytical Chemistry
• Chemometrics
• Spectroscopy

Background:

• Near-infrared (NIR) spectroscopy faces challenges with collinearity and non-influential variables.
• Variable selection is crucial for robust NIR model development.
• Weakly influential variables can improve model performance when combined effectively.

Purpose of the Study:

• To introduce a novel variable selection method, outlier removal with weight penalization and aggregation (OR-WPA).
• To enhance the accuracy and stability of NIR models by effectively utilizing all spectral variables.
• To improve the prediction of low-content components in complex samples.

Main Methods:

• OR-WPA identifies and removes outlier spectral variables based on coefficient of variation.
• It constructs submodels, assigns weights to variables based on regression coefficients, and applies a moving window for weight aggregation.
• Variables with excessively high weights are penalized to promote the inclusion of weakly influential variables.

Main Results:

• OR-WPA demonstrated superior predictive performance compared to existing methods on three diverse NIR datasets.
• The method significantly improved accuracy and stability, particularly for predicting low-content analytes.
• OR-WPA effectively balances the contribution of strongly and weakly influential spectral variables.

Conclusions:

• OR-WPA offers a robust and effective approach for variable selection in NIR spectroscopy.
• The method enhances model reliability and predictive power, especially in challenging applications.
• OR-WPA represents a significant advancement in chemometric modeling for spectral data analysis.