BartSmiles: Generative Masked Language Models for Molecular Representations.

Area of Science:

• Computational chemistry
• Machine learning
• Bioinformatics

Background:

• Self-supervised learning (SSL) is crucial for developing effective molecular representations.
• Generative masked language models (MLMs) show promise for molecular data.
• Existing SSL strategies for molecules have limitations.

Purpose of the Study:

• To develop a robust self-supervised strategy for molecular representations using generative MLMs.
• To train and evaluate BARTSmiles, a BART-like model, for molecular tasks.
• To demonstrate the effectiveness of learned representations for downstream applications.

Main Methods:

• Conducted in-depth ablations to refine the self-supervised pretraining strategy.
• Trained BARTSmiles using an order of magnitude more compute than prior methods.
• Evaluated BARTSmiles on diverse tasks: classification, regression, and generation.

Main Results:

• BARTSmiles consistently outperformed existing self-supervised representations.
• Achieved new state-of-the-art performance on eight benchmark tasks.
• Demonstrated that BART objective implicitly encodes downstream task information.
• Showcased that specific neurons in BARTSmiles predict molecular properties with high accuracy.
• Applied interpretability methods to identify chemically relevant substructures.

Conclusions:

• The proposed self-supervised strategy and BARTSmiles model represent a significant advancement in molecular representation learning.
• BARTSmiles offers a powerful and versatile tool for various molecular prediction and generation tasks.
• The model's interpretability provides valuable chemical insights, aiding scientific discovery.