Tapping on the Black Box: How Is the Scoring Power of a Machine-Learning Scoring Function Dependent on the Training...

Area of Science:

• Computational chemistry
• Drug discovery
• Machine learning in cheminformatics

Background:

• Machine-learning (ML) scoring functions are increasingly used for protein-ligand interactions.
• Concerns exist regarding the dependency of ML scoring function performance on training and test set overlap.
• Systematic evaluation is needed to understand ML model generalizability.

Purpose of the Study:

• To systematically assess the performance of six ML algorithms for protein-ligand scoring.
• To investigate the impact of training set size and similarity on ML scoring function accuracy.
• To compare ML scoring functions against conventional methods under varying data conditions.

Main Methods:

• Six ML algorithms (BRR, DT, KNN, MLP, L-SVR, RF) were trained on subsets of the PDBbind refined set (2016).
• Scoring functions were evaluated on the CASF-2016 test set.
• Training set size and similarity to the test set were systematically varied.
• Performance was compared to conventional scoring functions (ChemScore, ASP, X-Score).

Main Results:

• ML model performance, particularly Random Forest, was dependent on training set size and similarity.
• Conventional scoring functions showed consistent performance regardless of training set variations.
• Random Forest exhibited the best learning capability among the tested ML algorithms.
• "Soft overlap" between training and test sets is crucial for evaluating ML scoring functions.

Conclusions:

• The performance of ML scoring functions is sensitive to training data characteristics (size, similarity).
• Careful consideration of training-test set overlap is essential for robust evaluation of ML scoring functions.
• Standardized datasets accounting for similarity are proposed for future benchmarking.
• Random Forest demonstrates strong potential for developing accurate protein-ligand scoring functions.