Molecular language models: RNNs or transformer?

Area of Science:

• Computational Chemistry
• Machine Learning
• Drug Discovery

Background:

• Language models excel at learning molecular distributions for molecular generation.
• Recurrent Neural Networks (RNNs) were historically used for sequence data, while attention mechanisms and Transformer-Layer models have gained prominence.
• Understanding the comparative performance of RNNs and Transformer-Layers in molecular distribution learning is crucial.

Purpose of the Study:

• To investigate the differences between RNNs and Transformer-Layer models in learning complex molecular distributions.
• To evaluate their performance on diverse generative tasks, including penalized LogP, multimodal distributions, and large molecule datasets.
• To compare the efficacy of SMILES and SELFIES molecular representations.

Main Methods:

• Experimentation with three generative tasks: penalized LogP, multimodal distributions, and largest molecules from PubChem.
• Application of both Recurrent Neural Networks (RNNs) and Transformer-Layer models.
• Utilized two molecular representations: SMILES and SELFIES.
• Evaluation using molecular properties, basic metrics, and Tanimoto similarity.

Main Results:

• Both RNNs and Transformer-Layer models successfully learned complex molecular distributions.
• The SMILES representation demonstrated superior performance compared to SELFIES.
• RNNs are better suited for datasets emphasizing local features, while Transformer-Layers excel with global features and larger molecular weights.

Conclusions:

• Language models are effective for learning intricate molecular distributions.
• The choice between RNNs and Transformer-Layers should be guided by dataset characteristics, specifically the emphasis on local versus global features.
• SMILES representation offers advantages over SELFIES for these molecular generation tasks.