Critical Artifacts Improve Reproducibility of Protein-Ligand Binding Affinity Prediction Models on CASF-2016

Area of Science:

• Computational Chemistry
• Drug Discovery
• Bioinformatics

Background:

• Protein-ligand binding affinity prediction (PLBAP) models are vital tools in drug discovery.
• These models are commonly evaluated using the CASF-2016 dataset and Pearson correlation coefficient (PCC).
• Published PCC values are often used as benchmarks, assuming reproducibility.

Purpose of the Study:

• To systematically audit the reproducibility of 50 PLBAP models published between 2021-2024.
• To assess the feasibility of independently verifying reported CASF-2016 scoring power metrics.
• To introduce a community standard for PLBAP pipeline reproducibility.

Main Methods:

• Conducted a reproducibility audit of 50 PLBAP models using public resources.
• Attempted to reproduce CASF-2016 inference for each model.
• Introduced a five-item checklist for PLBAP pipeline reproducibility (license, code, weights).

Main Results:

• Only 17 out of 50 (34%) PLBAP pipelines met all checklist criteria for consistent runnability.
• Of the runnable models, only 9 (53%) achieved statistically reproducible results.
• Significant gaps were identified in the availability of code, documentation, and pretrained weights.

Conclusions:

• The assumption of reproducibility for published PLBAP models is often invalid.
• A significant portion of PLBAP models are not independently verifiable.
• The proposed checklist can serve as a community standard to improve PLBAP pipeline reproducibility.