Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset...

Area of Science:

• Computational biology
• Pharmacogenomics
• Machine learning in drug discovery

Background:

• Deep learning and machine learning models show potential for drug response prediction (DRP).
• However, their ability to generalize across different datasets is not well-established, limiting real-world applicability.
• Current evaluations lack standardized approaches, hindering reliable model comparison.

Purpose of the Study:

• To introduce a standardized benchmarking framework for evaluating cross-dataset generalization in DRP models.
• To develop novel evaluation metrics for assessing absolute and relative model performance across datasets.
• To provide a foundation for robust DRP model development and comparison.

Main Methods:

• Utilized five publicly available drug screening datasets and seven standardized DRP models.
• Developed a scalable workflow for systematic, cross-dataset evaluation.
• Introduced metrics to quantify absolute performance and performance drop on unseen data.

Main Results:

• Significant performance drops were observed when DRP models were tested on datasets different from their training data.
• No single model consistently outperformed others across all cross-dataset evaluations.
• The CTRPv2 dataset emerged as the most effective for training DRP models, leading to better generalization.

Conclusions:

• Rigorous assessment of cross-dataset generalization is crucial for DRP models.
• A standardized framework is essential for reliable evaluation and comparison of DRP models.
• Further development is needed to create DRP models with robust real-world applicability.