Exploring BERT for Reaction Yield Prediction: Evaluating the Impact of Tokenization, Molecular Representation, and...

Area of Science:

• Computational chemistry
• Machine learning in chemistry
• Synthetic chemistry

Background:

• Predicting reaction yields is crucial but challenging in synthetic chemistry.
• High-throughput experimentation (HTE) data provides valuable information for model training.
• BERT-based models show promise for reaction yield prediction.

Purpose of the Study:

• To systematically evaluate factors influencing BERT-based yield prediction models.
• To assess the impact of tokenization, molecular representation, and pretraining data.
• To investigate the effectiveness of adversarial training for improved model performance.

Main Methods:

• Utilized BERT-based models for yield prediction of Buchwald-Hartwig and Suzuki-Miyaura coupling reactions.
• Evaluated various tokenization methods (BPE, SentencePiece, WordPiece) and molecular representations (SMILES, DeepSMILES, SELFIES, etc.).
• Compared performance using different pretraining data sizes and explored artificially generated datasets, alongside adversarial training.

Main Results:

• Molecular representation had minimal impact; BPE and SentencePiece tokenization performed best.
• Smaller pretraining datasets (<100 K reactions) yielded comparable results to larger ones.
• Hybrid pretraining sets (real + artificial data) and adversarial training improved model robustness and generalizability.

Conclusions:

• Tokenization and pretraining strategies significantly impact ML model performance for chemical yield prediction.
• Artificially generated data can serve as an effective surrogate for real-world datasets.
• Adversarial training offers a novel approach to enhance the robustness and applicability of ML models in synthetic chemistry.