Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization.

Area of Science:

• Computational chemistry
• Machine learning in drug discovery
• Bioinformatics

Background:

• Overfitting is a common issue in predictive modeling, particularly when training and validation datasets share significant similarities.
• Ligand-based methods in drug discovery rely on chemical similarity to predict molecular activity, making them susceptible to training-validation redundancy.

Purpose of the Study:

• To introduce the Assessment of Validation-set Enrichment (AVE) score, a novel metric for quantifying training-validation redundancy in ligand-based classification.
• To evaluate the impact of AVE bias on the performance of various ligand-based predictive models across multiple benchmarks.

Main Methods:

• Development and application of the AVE score, which considers similarities among both active and inactive molecules.
• Testing AVE bias across seven established benchmarks commonly used for virtual screening and classification tasks.
• Analysis of the correlation between AVE bias and the performance of ligand-based predictive methods, independent of specific chemical representations or similarity metrics.

Main Results:

• A strong positive correlation was observed between the level of AVE bias and the reported performance of ligand-based predictive methods.
• This correlation persisted across different prediction tasks, chemical fingerprints, similarity measures, and existing bias-correction techniques.
• The findings suggest that previously reported high accuracies for many ligand-based methods might be inflated due to overfitting to benchmark datasets.

Conclusions:

• The AVE score provides a crucial measure for detecting and quantifying training-validation redundancy in ligand-based classification.
• A significant portion of the success attributed to ligand-based methods may be an artifact of overfitting to existing benchmarks.
• Future research should incorporate AVE or similar measures to ensure more reliable prospective accuracy assessments in drug discovery and cheminformatics.