Beware of machine learning-based scoring functions-on the danger of developing black boxes

Area of Science:

• Computational chemistry
• Cheminformatics
• Machine learning in drug discovery

Background:

• Machine learning (ML) algorithms using protein-ligand descriptors show promise for predicting binding constants.
• Recent studies suggest ML approaches outperform traditional empirical scoring functions.

Purpose of the Study:

• To validate the claimed superiority of ML-based scoring functions over empirical methods.
• To assess the performance of ML scoring functions in virtual screening enrichment.
• To investigate the sensitivity of ML scoring functions to ligand pose accuracy.

Main Methods:

• Trained Random Forest and Support Vector Machine models on protein-ligand distance counts from the PDBBind dataset.
• Evaluated scoring functions using 10 DUD-E datasets for virtual screening enrichment.
• Systematically varied ligand poses from X-ray coordinates to test pose sensitivity.

Main Results:

• ML scoring functions accurately predicted binding constants but failed to enrich virtual screening hit lists.
• Empirical Surflex-Dock showed sensitivity to docking pose quality, while ML models were insensitive.
• ML models, trained on basic distance counts, demonstrated insensitivity to ligand pose variations up to 10 Å RMSD.

Conclusions:

• While ML is valuable for scoring function design, using simple protein-ligand element-element distance counts requires caution.
• Proposed two essential benchmarking tests for novel scoring functions: pose sensitivity and virtual screening enrichment capability.
• Emphasized the need for meaningful application of ML descriptors to avoid developing ineffective scoring functions.