MolE: a foundation model for molecular graphs using disentangled attention.

Area of Science:

• Computational chemistry
• Machine learning in drug discovery
• Cheminformatics

Background:

• Accurate prediction of chemical properties from molecular structure is crucial in chemical sciences.
• Small training datasets limit the generalization ability of predictive models.
• Self-supervised pretraining on large unlabeled datasets followed by fine-tuning on labeled data has emerged as a solution.

Purpose of the Study:

• To introduce MolE, a Transformer architecture adapted for molecular graphs.
• To develop a two-step pretraining strategy for enhanced molecular property prediction.
• To improve the generalization performance of models in predicting ADMET properties.

Main Methods:

• Developed MolE, a Transformer architecture tailored for molecular graph representation.
• Implemented a two-step pretraining strategy: 1) self-supervised learning on ~842 million molecular graphs for chemical structure understanding, and 2) multi-task learning for biological information.
• Fine-tuned pretrained MolE models on smaller, labeled datasets for specific tasks.

Main Results:

• Fine-tuned MolE models achieved superior performance compared to existing state-of-the-art.
• Outperformed best published results on 10 out of 22 ADMET tasks in the Therapeutic Data Commons leaderboard.
• Demonstrated improved generalization capabilities due to the pretraining strategy.

Conclusions:

• The MolE architecture and its pretraining strategy significantly advance molecular property prediction.
• This approach offers a powerful tool for accelerating drug discovery and development.
• The findings highlight the potential of large-scale pretraining for cheminformatics applications.