Relationship between prediction accuracy and uncertainty in compound potency prediction using deep neural networks...

Area of Science:

• Molecular Machine Learning
• Computational Chemistry
• Drug Discovery

Background:

• Evaluating machine learning models requires assessing prediction variance or uncertainty.
• Uncertainty quantification (UQ) is an emerging research area in molecular machine learning with no standardized methods.
• Developing reliable UQ approaches is crucial for trustworthy AI in drug discovery.

Purpose of the Study:

• To analyze deep neural network variants and control models for compound potency prediction.
• To investigate the relationship between prediction accuracy and uncertainty in molecular machine learning.
• To identify potential biases and limitations in current UQ metrics.

Main Methods:

• Comparative analysis of diverse machine learning models (deep neural networks, control models).
• Evaluation of prediction accuracy and uncertainty metrics across different model complexities.
• Assessment of model performance across varying compound potency levels and training set modifications.

Main Results:

• Highly variable prediction uncertainties were observed for models with comparable accuracy using different UQ metrics.
• Model predictions and uncertainties showed strong dependence on the potency levels of test compounds.
• Deep neural networks exhibited inconsistent responses to training set changes, impacting uncertainty estimates.

Conclusions:

• Current UQ metrics show little to no correlation with prediction accuracy for compound potency prediction, particularly with deep neural networks.
• Existing UQ methods may lead to over- or under-confident predictions, highlighting the need for improved approaches.
• Further research is required to develop robust and standardized UQ guidelines for molecular machine learning applications.