In Need of Bias Control: Evaluating Chemical Data for Machine Learning in Structure-Based Virtual Screening

Area of Science:

• Computational Chemistry
• Pharmacology
• Artificial Intelligence

Background:

• Machine learning (ML) methods, including convolutional neural networks, are increasingly applied to structure-based virtual screening (SBVS).
• These ML models often outperform traditional empirical scoring functions in retrospective validation studies.
• However, ML models are frequently treated as 'black boxes,' lacking interpretability regarding the features driving predictions.

Purpose of the Study:

• To re-evaluate the suitability of widely used benchmark datasets for ML-based SBVS.
• To identify and demonstrate implicit biases learned from standard benchmarks in ML models.
• To advocate for improved validation strategies and suitable benchmark datasets for ML in SBVS.

Main Methods:

• Re-evaluation of three established benchmark datasets commonly used for ML in SBVS.
• Analysis of two current literature examples to illustrate implicit bias acquisition in ML models.
• Development of guidelines for designing unbiased validation experiments for ML-based SBVS.

Main Results:

• Not all standard benchmark datasets are appropriate for validating ML methods in SBVS.
• Implicit biases can be learned unnoticed from conventional benchmarks, potentially compromising model reliability.
• Existing benchmarks may not adequately control for biases, leading to misleading performance assessments.

Conclusions:

• There is a critical need for eligible validation experiments and ML-suited benchmark datasets for unbiased SBVS.
• Developing new, bias-controlled benchmark datasets is essential for advancing reliable ML applications in drug discovery.
• Guidelines are provided for setting up robust validation experiments and generating appropriate datasets.