Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Prediction Errors for Deep Neural...

Area of Science:

• Computational chemistry
• Machine learning in drug discovery
• Bioinformatics

Background:

• Deep learning models are increasingly used for in vitro compound activity prediction in drug discovery.
• Estimating the reliability and confidence of these deep learning predictions is crucial but underexplored.
• Current methods lack robust techniques for quantifying prediction uncertainty in virtual screening.

Purpose of the Study:

• To introduce Deep Confidence, a novel framework for calculating valid and efficient confidence intervals for individual predictions from deep learning models.
• To enhance the trustworthiness and interpretability of virtual screening models in drug discovery.
• To integrate Snapshot Ensembling and conformal prediction for robust error estimation.

Main Methods:

• Implemented Snapshot Ensembling to create an ensemble of deep neural networks by capturing parameters at local minima during training.
• Utilized conformal prediction framework, leveraging variability across base learners and validation residuals to compute confidence intervals.
• Applied the Deep Confidence framework to 24 diverse IC50 datasets from ChEMBL 23.

Main Results:

• Snapshot Ensembles demonstrated performance comparable to Random Forest and independently trained deep neural network ensembles.
• The Deep Confidence framework produced narrower confidence intervals compared to existing methods.
• The framework successfully generated valid and efficient confidence estimates for predictions.

Conclusions:

• Deep Confidence offers a versatile and computationally efficient method for estimating prediction reliability in deep learning applications.
• The framework significantly improves the interpretability and trustworthiness of virtual screening models.
• This approach can be readily applied to various deep learning-based drug discovery tasks without additional computational expense.