How evaluation choices distort the outcome of generative drug discovery

"How to evaluate the de novo designs proposed by a generative model?" Despite the transformative potential of generative deep learning in drug discovery, this seemingly simple question has no clear answer. The absence of standardized guidelines challenges both the benchmarking of generative approaches and the selection of molecules for prospective studies. In this work, we take a fresh - critical and constructive - perspective on de novo design evaluation. By training chemical language models, we analyze approximately 1 billion molecule designs and discover principles consistent across different neural networks and datasets. We uncover a key confounder: the size of the generated molecular library significantly impacts evaluation outcomes, often leading to misleading model comparisons. We find increasing the number of designs as a remedy and propose new and compute-efficient metrics to compute at large-scale. We also identify critical pitfalls in commonly used metrics - such as uniqueness and distributional similarity - that can distort assessments of generative performance. To address these issues, we propose new and refined strategies for reliable model comparison and design evaluation. Furthermore, when examining molecule selection and sampling strategies, our findings reveal the constraints to diversify the generated libraries and draw new parallels and distinctions between deep learning and drug discovery. We anticipate our findings to help reshape evaluation pipelines in generative drug discovery, paving the way for more reliable and reproducible generative modeling approaches. SCIENTIFIC CONTRIBUTION: Our work takes a step toward enhancing the robustness and reliability of evaluation practices in generative drug discovery. We systematically analyze current evaluation practices using approximately one billion designs from deep learning models. We find that the number of designs, often an overlooked parameter, can distort scientific outcomes related to distributional similarity and diversity. Moreover, we show that using larger design libraries than are typically adopted helps to avoid this pitfall, and we develop efficient algorithms to enable large-scale studies. We also propose guidelines for prospective molecule selection and uncover inherent constraints in diversifying molecular designs.