Machine Learning for De Novo Molecular Generation: A Comprehensive Review

Deep generative models have emerged as powerful computational engines for de novo molecular design, enabling efficient exploration of a vast chemical space that remains inaccessible to traditional experimental approaches. This review provides a comprehensive survey of machine learning-driven molecular generation, systematically organizing the field across three foundational pillars: molecular representations, model architectures, and evaluation frameworks. We present a detailed taxonomy of state-of-the-art generative models, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs), Transformers, Diffusion Models, Normalizing Flows, and Hybrid Architectures, analyzing their underlying mechanisms, comparative strengths, and inherent limitations. Critically, we depart from purely descriptive surveys by systematically examining algorithmic failure modes and practical deployment challenges across model families. We discuss core applications spanning distribution learning and goal-directed generation. Special attention is given to challenging therapeutic domains such as Central Nervous System (CNS) drug discovery, where stringent constraints like blood-brain barrier (BBB) permeability and neurotoxicity mitigation demand multiparameter optimization. We critically evaluate the gap between computational benchmarks and practical medicinal chemistry, addressing synthetic feasibility and experimental validation. Subsequently, we highlight persistent theoretical, computational, and empirical challenges that currently limit widespread deployment, and outline promising future opportunities, including physics-informed architectures, large language models, and autonomous laboratories. This review aims to provide actionable insights for both machine learning researchers and medicinal chemists engaged in next-generation drug discovery.