Related Experiment Video
Updated: Jul 8, 2026
10:41
Leveraging CyVerse Resources for De Novo Comparative Transcriptomics of Underserved (Non-model) Organisms
Published on: May 9, 2017
ScrambleBench: a workflow for comparative assessment of structure-based de novo generative models
Veincent Yap1, Pan Xu1, Frankie S Mak1
1Experimental Drug Development Centre, Agency for Science, Technology and Research (A*STAR), 10 Biopolis Road, Chromos, Singapore, 138670, Singapore.
Journal of Cheminformatics
|July 7, 2026
Summary
Generative AI for drug discovery needs standardized evaluation. ScrambleBench offers a robust workflow to assess AI models, revealing limitations in generalization and highlighting the need for improved chemical diversity and binding accuracy in de novo molecular design.
Area of Science:
- Computational chemistry and drug discovery.
- Artificial intelligence in molecular design.
Background:
- Generative artificial intelligence (AI) shows promise for de novo small molecule design in drug discovery.
- Existing AI models for drug design often lack standardized evaluation, hindering reliable integration into medicinal chemistry workflows.
- Assessing the robustness and reliability of structure-based generative AI models is crucial for their practical application.
Purpose of the Study:
- To introduce ScrambleBench, a standardized benchmarking workflow for evaluating structure-based generative AI models in drug discovery.
- To assess the performance of six representative generative AI models across diverse protein targets.
- To identify key areas for improvement in AI-driven molecular design, focusing on chemical diversity, binding conformation, and docking affinity.
Main Methods:
- Developed ScrambleBench, a unified workflow integrating diversity analysis, conformational validity, docking reproducibility, pharmacophore matching, and virtual hit rate.
- Evaluated six generative AI models (Pocket2Mol, PocketFlow, Lingo3DMol, DiffSBDD, PMDM, Chemistry42) against GPCRs, kinases, and hydrolases.
- Utilized metrics including Hamiltonian Diversity (HamDiv) for assessing molecular set quality and dissimilarity.
Main Results:
- No single generative AI model demonstrated overall dominance across all evaluated criteria.
- Models exhibited limited generalization to target binding sites, even for proteins present in training data, resulting in high redocking RMSD and low virtual hit rates.
- Explicit evaluation of chemical diversity and improved loss functions emphasizing physicochemical properties and pharmacophore recognition are necessary.
Conclusions:
- ScrambleBench provides a transparent and reproducible framework for evaluating structure-based generative AI models in drug discovery.
- Current generative models show limitations in generalization and require enhancements for practical medicinal chemistry applications.
- Future AI frameworks should prioritize drug-like properties, accurate pharmacophore recognition, and robust chemical diversity assessment.
