Using beta binomials to estimate classification uncertainty for ensemble models.

Area of Science:

• Computational chemistry and cheminformatics
• Pharmacology and toxicology

Background:

• Quantitative structure-activity relationship (QSAR) models offer significant potential for reducing drug discovery costs and animal testing.
• Estimating the overall reliability of QSAR models has advanced, but assessing confidence in individual predictions remains crucial for researchers and regulators.

Purpose of the Study:

• To develop a method for estimating the confidence in individual predictions made by ensemble models.
• To leverage the agreement among submodels within an ensemble to quantify prediction reliability.

Main Methods:

• Utilized artificial neural network ensembles (ANNEs) with two classification methods: vote tallies and averaged outputs.
• Modeled prediction and error distributions using the beta binomial distribution.
• Validated the method on large datasets including logP, Ames mutagenicity, and CYP2D6 inhibition data.

Main Results:

• The agreement among submodels in an ensemble provides information on prediction reliability.
• Beta binomial distributions accurately modeled prediction and error distributions, enabling probability estimation of classification errors.
• The method accurately predicted external validation set performance using training pool data, even with imbalanced datasets.
• Prospective misclassification likelihood correlated with submodel consensus and was accurately estimated.

Conclusions:

• Confidence in individual ensemble predictions can be accurately assessed by analyzing prediction and error distributions relative to submodel agreement.
• Ensemble uncertainty estimation can be improved by adjusting classification thresholds based on error distribution parameters.
• Model profiles can indicate unreliable uncertainty estimates without external validation.