Large-Scale Distributed Training of Transformers for Chemical Fingerprinting.

Area of Science:

• Computational Chemistry
• Machine Learning
• Artificial Intelligence

Background:

• Transformer models are powerful for machine learning but require substantial data and compute.
• Their application in chemistry includes reaction classification and property prediction.
• Distributed computing offers a viable solution for training these models in academic settings.

Purpose of the Study:

• To demonstrate the feasibility of training transformer models for chemical tasks using distributed computing.
• To introduce MFBERT (Molecular Fingerprints through Bidirectional Encoder Representations from Transformers).
• To achieve state-of-the-art performance in generating molecular fingerprints for virtual screening.

Main Methods:

• Pre-training a transformer model using distributed computing on a large chemical literature dataset.
• Utilizing a SentencePiece tokenization model for a data-driven approach.
• Fine-tuning the model on specific datasets for targeted fingerprint generation.

Main Results:

• Achieved state-of-the-art scores on a virtual screening benchmark for molecular fingerprints.
• Successfully generated targeted molecular fingerprints through fine-tuning.
• Demonstrated a data-driven, tokenization-free process from raw molecular data to fingerprints.

Conclusions:

• Distributed computing enables effective training of transformer models for complex chemical problems.
• MFBERT provides a powerful, data-driven method for generating high-quality molecular fingerprints.
• The approach enhances virtual screening and other chemical machine learning applications.