Transformer technology in molecular science

A transformer is the foundational architecture behind large language models designed to handle sequential data by using mechanisms of self-attention to weigh the importance of different elements, enabling efficient processing and understanding of complex patterns. Recently, transformer-based models have become some of the most popular and powerful deep learning (DL) algorithms in molecular science, owing to their distinctive architectural characteristics and proficiency in handling intricate data. These models leverage the capacity of transformer architectures to capture complex hierarchical dependencies within sequential data. As the applications of transformers in molecular science are very widespread, in this review, we only focus on the technical aspects of transformer technology in molecule domain. Specifically, we will provide an in-depth investigation into the algorithms of transformer-based machine learning techniques in molecular science. The models under consideration include generative pre-trained transformer (GPT), bidirectional and auto-regressive transformers (BART), bidirectional encoder representations from transformers (BERT), graph transformer, transformer-XL, text-to-text transfer transformer, vision transformers (ViT), detection transformer (DETR), conformer, contrastive language-image pre-training (CLIP), sparse transformers, and mobile and efficient transformers. By examining the inner workings of these models, we aim to elucidate how their architectural innovations contribute to their effectiveness in processing complex molecular data. We will also discuss promising trends in transformer models within the context of molecular science, emphasizing their technical capabilities and potential for interdisciplinary research. This review seeks to provide a comprehensive understanding of the transformer-based machine learning techniques that are driving advancements in molecular science.