Positional embeddings and zero-shot learning using BERT for molecular-property prediction.

Area of Science:

• Cheminformatics
• Deep Learning
• Natural Language Processing

Background:

• Advancements in cheminformatics necessitate improved handling of chemical simplified molecular input line entry system (SMILES) data.
• Transformer models, successful in natural language processing, show promise for molecular data analysis.
• Positional encodings (PEs) are crucial for transformer models to capture sequential and contextual information in molecular representations.

Purpose of the Study:

• To explore and optimize various positional encodings (PEs) within a transformer-based framework for enhanced molecular property prediction.
• To investigate the impact of PEs on the accuracy and generalization of Bidirectional Encoder Representations from Transformer (BERT) models for chemical text analysis using SMILES and DeepSMILES.
• To assess the zero-shot learning capabilities of BERT models with different PEs on diverse molecular datasets.

Main Methods:

• Pretraining Bidirectional Encoder Representations from Transformer (BERT) models with various positional encodings (PEs) on SMILES strings.
• Fine-tuning the best-performing BERT models on downstream molecular property prediction tasks.
• Utilizing both SMILES and DeepSMILES representations for comprehensive evaluation across existing and newly proposed datasets.

Main Results:

• Demonstrated improved accuracy and generalization in molecular property prediction through optimized PEs in BERT models.
• Successfully applied BERT models with various PEs to diverse datasets, including COVID-19 and bioassay data.
• Showcased the model's proficiency in zero-shot learning for predicting properties of unseen molecular representations.

Conclusions:

• Positional encodings significantly enhance the performance of transformer models like BERT for molecular property prediction.
• BERT models, when equipped with appropriate PEs, exhibit robustness and potential for broad applications in cheminformatics and bioinformatics.
• The study highlights the effectiveness of PEs in capturing complex atomic relationships within SMILES strings for accurate chemical data analysis.