Entropy-based byte patching transformer for self-supervised pretraining of SMILES data

Area of Science:

• Computational Chemistry
• Machine Learning
• Bioinformatics

Background:

• Transformer models are advancing molecular representation learning.
• Capturing localized and hierarchical chemical structures remains a challenge for current models.

Purpose of the Study:

• To introduce the SMILES Byte-Patch Transformer (SMiBPT) for improved molecular representation learning.
• To develop an adaptive model for dynamic segmentation of chemical strings.

Main Methods:

• SMiBPT uses entropy-based byte patching to segment SMILES and DeepSMILES strings into chemically meaningful substructures.
• The model integrates self-supervised pretraining, chemical motif-aware encoding, adaptive entropy-aware masking, and rotary position embeddings.
• Trained on ~216 million unlabeled molecules from PubChem without truncation.

Main Results:

• SMiBPT outperforms existing models like ChemBERTa, SMILES-BERT, and MoLFormer in predictive accuracy and efficiency.
• The adaptive patching strategy preserves molecular semantics and enhances feature extraction.
• Demonstrated superior zero-shot transfer capabilities.

Conclusions:

• SMiBPT offers a parameter-efficient and effective approach to molecular representation learning.
• The adaptive segmentation method addresses limitations of fixed tokenization in chemical learning.
• This model advances the application of large language models in chemistry.