UNIQUE: A Framework for Uncertainty Quantification Benchmarking.

Area of Science:

• Computational chemistry
• Data science
• Drug discovery

Background:

• Machine learning (ML) models are integral to decision-making in fields like drug discovery.
• Evaluating ML model robustness and predictive power in real-world scenarios is challenging.
• Uncertainty quantification (UQ) is vital for assessing ML model reliability, but a universal strategy is lacking.

Purpose of the Study:

• To introduce the UNIQUE (UNcertaInty QUantification bEnchmarking) framework for comparing UQ strategies in ML.
• To provide a standardized approach for evaluating UQ methodologies across diverse applications.
• To facilitate the development and assessment of new UQ methods.

Main Methods:

• Development of the UNIQUE Python library for unifying UQ metric benchmarking.
• Implementation of standard and nonstandard UQ metrics, integrating dataset and model information.
• Evaluation of UQ metrics across various application scenarios, including confidence-based filtering and acquisition functions.

Main Results:

• The UNIQUE framework enables comprehensive benchmarking of multiple UQ strategies.
• It allows for the calculation of novel UQ metrics tailored to specific applications.
• The library facilitates a consistent and thorough evaluation of UQ performance.

Conclusions:

• The UNIQUE framework standardizes UQ investigations in ML-based predictions.
• It aids in selecting appropriate UQ metrics for different tasks and datasets.
• This tool will advance the reliability and applicability of ML models in scientific research.