Insights into performance evaluation of compound-protein interaction prediction methods

Area of Science:

• Computational chemistry and cheminformatics
• Drug discovery and development
• Machine learning in bioinformatics

Background:

• Machine-learning-based prediction of compound-protein interactions (CPIs) is crucial for drug design, screening, and repurposing.
• Existing studies often claim improved predictive accuracy but may suffer from overoptimistic performance estimates due to experimental design flaws.

Purpose of the Study:

• To systematically analyze factors affecting the generalization performance of CPI predictors, including cross-validation similarity, negative example synthesis, and evaluation protocol alignment.
• To identify robust experimental design strategies for accurate CPI prediction applicable to drug repurposing and ligand discovery.

Main Methods:

• Analysis of CPI predictor generalization performance using state-of-the-art methods and a kernel-based baseline.
• Systematic evaluation of factors: training-test similarity, synthetic negative example generation strategies (random pairing vs. sophisticated), and alignment of evaluation metrics with real-world screening.
• Implementation of stringent performance assessment protocols.

Main Results:

• Effective assessment of CPI predictor generalization requires careful control over training and test set similarity.
• A simple kernel-based approach, under stringent assessment, outperformed existing state-of-the-art methods.
• Random pairing for synthetic negative examples yielded better generalization than more sophisticated strategies.
• Proposed strategies can significantly improve CPI prediction for screening and identifying ligands for SARS-CoV-2-Spike and Human-ACE2.

Conclusions:

• Current experimental designs for CPI prediction often lead to overoptimistic performance estimates.
• Stringent validation protocols and careful consideration of training-test similarity are essential for reliable CPI prediction.
• Simple, well-validated methods can achieve superior performance, offering practical improvements for drug discovery and repurposing.