MolDeBERTa: Foundational Model for Physicochemical and Structural-Informed Molecular Representation Learning.

Area of Science:

• * Computational Chemistry
• * Cheminformatics
• * Machine Learning

Background:

• * Foundational models are crucial for accelerating material and drug discovery.
• * Existing molecular language models use generic objectives, neglecting physicochemical properties.
• * There's a need for structure-informed models that bridge linguistic and physical molecular representations.

Purpose of the Study:

• * Introduce MolDe-BERTa, a structure-informed self-supervised molecular encoder.
• * Develop novel pretraining objectives to embed molecular properties into latent space.
• * Advance unsupervised encoder-based foundational models for chemistry-informed representation learning.

Main Methods:

• * Utilized a byte-level Byte-Pair Encoding (BPE) tokenization strategy.
• * Pretrained MolDe-BERTa on a large corpus of 123 million SMILES molecules from PubChem.
• * Introduced three novel pretraining objectives to bias towards molecular properties and structural similarity.

Main Results:

• * MolDe-BERTa outperformed existing masked language models on 9 downstream MoleculeNet benchmarks.
• * Achieved up to a 16% reduction in regression error.
• * Demonstrated improvements of up to 3.0 ROC-AUC points in classification tasks.

Conclusions:

• * MolDe-BERTa represents a significant advancement in unsupervised, chemistry-informed representation learning.
• * The model enables data-efficient learning by integrating structural and property information.
• * Publicly available code and datasets facilitate further research in molecular discovery.