A survey of transformer and LLM-based architectures, workflows, and systems in drug discovery

Drug discovery has greatly benefited from the recent progress in Artificial Intelligence (AI). Numerous tasks within pharmaceutical research and development are being formulated as classification, regression, generation, or sequence-to-sequence problems, and tackled via AI systems that can learn from data, making the process of discovery and screening faster and more efficient. Two of the greatest breakthroughs in AI have arguably been the advent of transformers and Large Language Models (LLMs). With their parallel processing capabilities, their ability to detect dependencies and interactions in sequence data, and the vast knowledge stored in their millions or billions of parameters, transformers and LLMs are successful, highly popular additions to the computational drug discovery arsenal. This work provides an extensive overview of transformer-based architectures and workflows utilized in drug discovery. The conducted analysis focuses on the types of transformers and LLMs currently in use, as well as the systems and workflows that integrate them. Research findings indicate the transformer architecture's exceptional flexibility and adaptability, which support its implementation across diverse configurations and systems. This versatility positions it as a transformative model with the promising potential to revolutionize drug discovery and drive significant scientific breakthroughs.