A novel importance scores based variable selection approach and validation using a MIR and NIR dataset.

Area of Science:

• Spectral analysis
• Chemometrics
• Machine learning

Background:

• Variable selection is crucial for accurate spectral data interpretation.
• Existing methods may not optimally identify the most informative spectral variables.

Purpose of the Study:

• Introduce a novel six-step variable selection process named VMHBSC.
• Demonstrate the effectiveness of VMHBSC in improving spectral analysis model performance.
• Evaluate VMHBSC on both mid-infrared (MIR) and near-infrared (NIR) spectral datasets.

Main Methods:

• Developed and applied the VMHBSC variable selection process.
• Utilized Decision Trees (DT), Gradient Boosting Decision Tree (GBDT), and Extreme Gradient Boosting (XGBoost) for model building.
• Tested VMHBSC on a MIR dataset of Chenpi samples and an NIR dataset for a modeling competition.

Main Results:

• VMHBSC identified 3 key variables from 7468 in the MIR dataset, significantly improving DT, GBDT, and XGBoost model accuracy.
• VMHBSC selected 24 important variables from 256 in the NIR dataset.
• Hybrid models (VMHBSC-DT, VMHBSC-GBDT, VMHBSC-XGBoost) demonstrated stable performance using the selected NIR variables.

Conclusions:

• The VMHBSC process is effective in enhancing model performance and robustness in spectral analysis.
• VMHBSC offers a powerful approach for variable selection in complex spectral datasets.
• This method improves the interpretability and accuracy of chemometric models.