Reliable Model Selection without Reference Values by Utilizing Model Diversity with Prediction Similarity.

Area of Science:

• Chemometrics
• Machine Learning
• Data Science

Background:

• Predictive modeling using diverse data (e.g., NIR spectra, QSAR) requires robust model selection for accurate calibration, training, and maintenance.
• Current model selection protocols often rely on single quality measures and cross-validation, which can be complex and unreliable for dynamic modeling tasks.

Purpose of the Study:

• To develop a generic, reliable model selection process for predictive modeling and spectral model maintenance.
• To introduce a novel consensus filtering approach that addresses the limitations of existing model selection methods.

Main Methods:

• Developed a consensus filtering approach (MDPS) prioritizing model diversity (MD) and prediction similarity (PS).
• Integrated a bias-variance trade-off measure into the model selection framework.
• Utilized unlabeled samples for active predictions, eliminating the need for cross-validation schemes.

Main Results:

• Demonstrated the versatility and reliability of the MDPS approach across four NIR datasets and one QSAR dataset.
• Showcased the ability of MDPS to select optimal tuning parameters for accurate prediction of new samples or molecules.
• Substantiated the Rashomon effect, indicating that multiple model tuning parameter values can yield accurate predictions.

Conclusions:

• The developed MDPS model selection process offers a significant advancement for predictive modeling and dynamic modeling tasks.
• MDPS provides a reliable and versatile protocol for selecting optimal models, even with unlabeled data.
• The findings highlight the importance of considering model diversity and prediction similarity for robust model selection.