Challenging Reaction Prediction Models to Generalize to Novel Chemistry.

Area of Science:

• Organic Chemistry
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning models are increasingly used for predicting organic reaction products, aiding retrosynthesis and molecular design.
• Existing benchmarks often evaluate models in-distribution, which may not reflect real-world performance.
• Current models can produce erroneous predictions when applied to novel or out-of-distribution scenarios.

Purpose of the Study:

• To evaluate the out-of-distribution generalization capabilities of a prototypical SMILES-based deep learning reaction predictor.
• To identify the limitations of current reaction prediction models in realistic deployment settings.
• To establish a framework for assessing model performance beyond standard benchmarks.

Main Methods:

• Evaluated a SMILES-based deep learning model on challenging out-of-distribution tasks.
• Assessed performance on data from new patents and authors, distinct from training data.
• Conducted time-split evaluations using reactions published years after the training set.
• Examined model extrapolation across different reaction classes.

Main Results:

• Performance on randomly sampled datasets is overly optimistic compared to generalization to new patents or authors.
• Models show decreased performance when tested on reactions published significantly later than the training data.
• Extrapolation across reaction classes reveals limitations in predicting novel reaction types.

Conclusions:

• Current deep learning models for reaction prediction exhibit significant limitations in out-of-distribution settings.
• Standard benchmarks do not adequately capture the challenges of real-world applications and extrapolation.
• Further development is needed to create next-generation models capable of robust reaction discovery and prediction.