UMAP-based clustering split for rigorous evaluation of AI models for virtual screening on cancer cell lines

Area of Science:

• Computational Chemistry and Cheminformatics
• Artificial Intelligence in Drug Discovery
• Machine Learning for Molecular Property Prediction

Background:

• Virtual Screening (VS) using Artificial Intelligence (AI) is vital for early drug discovery.
• Effective AI model benchmarking relies on appropriate data splitting strategies.
• Traditional methods like random, scaffold, and Butina splits can overestimate model performance due to unrealistic data partitioning.

Purpose of the Study:

• To evaluate the effectiveness of different data splitting methods for benchmarking AI models in molecular property prediction and VS.
• To compare traditional splitting techniques (random, scaffold, Butina clustering) against Uniform Manifold Approximation and Projection (UMAP) clustering.
• To provide a more realistic evaluation framework for AI models used in drug discovery.

Main Methods:

• Four datasets (NCI-60) were split using four methods: random, scaffold, Butina clustering, and UMAP clustering.
• Four representative AI models (Linear Regression, Random Forest, Transformer-CNN, GEM) were trained and evaluated.
• A total of 8400 models were trained and evaluated across all datasets and splitting methods.

Main Results:

• UMAP clustering provided the most challenging and realistic benchmarks for AI model evaluation.
• The performance ranking of splitting methods was UMAP > Butina > scaffold ≈ random.
• The study highlights the misalignment of ROC AUC with Virtual Screening goals, recommending context-specific metrics.

Conclusions:

• UMAP clustering is recommended as a superior data splitting method for molecular property prediction and VS, offering more reliable AI model benchmarking.
• Traditional methods like scaffold and Butina splits can lead to overestimated performance and should be used with caution.
• The findings advocate for the adoption of UMAP splits and appropriate evaluation metrics to advance the field of AI-driven drug discovery.