Diversity Beats Size Scaling for Chemical Language Models

Area of Science:

• Computational chemistry
• Artificial intelligence in drug discovery

Background:

• Chemical language models (e.g., transformers on SMILES) are vital in drug design.
• Rapid growth in model and dataset size has occurred, but performance impact is unclear.

Purpose of the Study:

• To systematically evaluate the effect of model and dataset size on transformer performance for molecular representations.
• To explore strategies for improving hit generation and diversity in drug discovery.

Main Methods:

• Trained encoder-decoder transformers on paired textual molecular representations.
• Systematically varied model size and dataset size.
• Introduced a dataset diversification strategy.

Main Results:

• Model scaling beyond a threshold did not improve hit generation rate.
• Dataset scaling showed diminishing returns.
• Dataset diversification significantly increased hit diversity.

Conclusions:

• Data curation and diversity are more critical than continued scaling for molecular hit discovery.
• A shift towards diversity-first training paradigms is motivated.