An open-source family of large encoder-decoder foundation models for chemistry

Area of Science:

• Computational chemistry and cheminformatics
• Machine learning applications in drug discovery
• Development of artificial intelligence for molecular modeling

Background:

• Foundation models, initially successful in natural language processing, are now being adapted for molecular modeling.
• Large-scale pre-training of chemical language models is crucial for effective representation learning across various tasks.
• Existing methods for molecular property estimation and reaction prediction can be improved with advanced AI techniques.

Purpose of the Study:

• To introduce a novel family of encoder-decoder chemical foundation models.
• To pre-train these models on an extensive dataset of molecular sequences.
• To evaluate their performance on property estimation and reaction outcome prediction tasks.

Main Methods:

• Development of encoder-decoder foundation models for molecular sequences.
• Pre-training on a curated dataset of 91 million molecular sequences from PubChem.
• Evaluation of model performance on benchmark datasets for property estimation and reaction prediction.
• Analysis of learned molecular representations and their properties.

Main Results:

• The proposed chemical foundation models match or surpass existing state-of-the-art approaches.
• Model variants demonstrate strong performance in property estimation and reaction outcome prediction.
• Learned molecular representations exhibit chemically relevant features and support few-shot learning.
• Decoder-based reconstruction objective is key to the learned embedding space structure.

Conclusions:

• The developed foundation models offer a general-purpose solution for molecular analysis and reasoning.
• These models require minimal supervision, making them highly adaptable.
• The findings highlight the potential of large-scale pre-training for advancing molecular intelligence.