Transformer-based models for chemical SMILES representation: A comprehensive literature review

Area of Science:

• Cheminformatics and Bioinformatics
• Natural Language Processing (NLP)
• Machine Learning (ML)
• Deep Learning (DL)

Background:

• Chemical Language Models (CLMs) are gaining traction, mirroring successes in NLP.
• Vast unlabeled chemical data necessitates CLMs with advanced reasoning.
• Molecular graphs and descriptors are traditional ML representations; Transformers offer a new paradigm.

Purpose of the Study:

• To review the state-of-the-art Transformer-based CLMs in chemical informatics.
• To analyze the capabilities of CLMs for tasks like de novo design and property prediction.
• To identify current limitations, challenges, and advantages of these models.

Main Methods:

• Review of Transformer-based language models (e.g., BERT, GPT variants) applied to chemical data.
• Analysis of models processing chemical SMILES strings for contextual information.
• Evaluation of CLMs in cheminformatics downstream tasks.

Main Results:

• Transformer-based CLMs demonstrate significant performance in NLP-inspired chemical tasks.
• These models effectively learn contextual information from SMILES sequences.
• Existing CLMs show promise for molecular property prediction and de novo design.

Conclusions:

• Transformer-based CLMs represent a powerful advancement in chemical informatics.
• Further research is needed to address limitations and explore future opportunities.
• The potential for CLMs in drug discovery and molecular design is substantial.