Explicit Applicability Domain Calculations Can Help Determine When Uncertainty Estimates Are Less Reliable

Area of Science:

• Computational Chemistry
• Cheminformatics
• Machine Learning in Drug Discovery

Background:

• Quantitative Structure-Activity Relationship (QSAR) models are crucial for predicting chemical compound properties.
• Uncertainty estimation in QSAR is vital for reliable predictions, with Conformal Regression and Venn-ABERS being state-of-the-art methods.
• These methods' performance degrades when applied to data distributions different from the training set, particularly in non-random splits like temporal or cluster validation.

Purpose of the Study:

• To investigate if explicit applicability domain calculations can improve the reliability of uncertainty estimates for QSAR models.
• To determine if structural similarity can predict when uncertainty estimates are less dependable for out-of-domain molecules.
• To assess the effectiveness of the k-nearest neighbors applicability domain approach (nUNC) in conjunction with uncertainty estimation methods.

Main Methods:

• Compared different applicability domain and uncertainty estimation methods using exemplar datasets.
• Extensively investigated the implications of applicability domain status on uncertainty estimation reliability using a k-nearest neighbors (nUNC) approach.
• Combined nUNC with Cross-Venn-ABERS Predictors (classification) and Aggregated Conformal Prediction (regression) on various public and industrial datasets, focusing on non-random temporal and cluster splits.

Main Results:

• Explicit applicability domain calculations using structural similarity effectively identify when uncertainty estimates are less reliable for out-of-domain predictions.
• The nUNC approach, combined with uncertainty estimation methods, demonstrated the ability to distinguish between reliable (in-domain) and less reliable (out-of-domain) uncertainty estimates.
• These findings were consistent across multiple public and industrial datasets, including temporal splits, highlighting practical applicability.

Conclusions:

• Integrating structural similarity-based applicability domain assessments with uncertainty estimation methods significantly enhances the trustworthiness of QSAR predictions.
• The nUNC method is a valuable tool for determining the reliability of uncertainty estimates, particularly for molecules outside the model's training distribution.
• This approach is crucial for real-world applications where QSAR models encounter diverse chemical spaces.