On the difficulty of validating molecular generative models realistically: a case study on public and proprietary...

Area of Science:

• * Computational chemistry and cheminformatics.
• * Artificial intelligence in drug discovery.

Background:

• * Validating deep generative models for de novo molecule design lacks established best practices.
• * Retrospective validation can be biased, while prospective validation is costly and subject to human selection bias.
• * Evaluating generative models requires methods that mimic real-world drug design processes.

Purpose of the Study:

• * To assess if a generative model trained on early-stage compounds can generate middle/late-stage compounds de novo.
• * To frame retrospective validation as the ability to mimic human drug design.
• * To investigate the performance of generative models in real-world drug discovery projects.

Main Methods:

• * Utilized experimental data from five public and six in-house drug discovery projects.
• * Pre-processed time-series data to reflect realistic synthetic expansions.
• * Employed REINVENT, a recurrent neural network (RNN)-based generative model.
• * Split datasets and trained REINVENT on early-stage compounds.

Main Results:

• * Middle/late-stage compound rediscovery rates were significantly higher in public projects (e.g., 1.60% in top 100) compared to in-house projects (0.00%).
• * Nearest neighbor similarity analysis revealed distinct patterns between active and inactive compounds in public versus in-house projects.
• * Generative models recovered few middle/late-stage compounds, indicating a gap between algorithmic design and practical drug discovery.

Conclusions:

• * Generative models recover a limited number of middle/late-stage compounds from real-world drug discovery projects.
• * A fundamental difference exists between purely algorithmic de novo design and the complexities of real-world drug discovery.
• * Retrospective evaluation of de novo compound design approaches is currently difficult or potentially impossible.