Optimizing SMILES token sequences via trie-based refinement and transition graph filtering.

Area of Science:

• Computational chemistry
• Machine learning for drug discovery
• Bioinformatics

Background:

• SMILES tokenization is crucial for molecular foundation models.
• Existing methods like SPE and APE struggle with chemical coherence and generalization.
• Ineffective tokenization leads to fragmented substructures and hinders model performance.

Purpose of the Study:

• To develop a domain-aware SMILES compression method for improved molecular representation learning.
• To enhance the efficiency and interpretability of tokenization for foundation models.
• To create compact and chemically meaningful token sequences.

Main Methods:

• A domain-aware method combining frequency-guided substring mining using a prefix trie.
• Optional entropy-based refinement using a token transition graph (TTG).
• Evaluation on PubChem and peptide corpora, including QSAR regression on ESOL.

Main Results:

• Trie+TTG reduced token sequences by over 50% compared to APE on PubChem, preserving chemical substructures.
• Achieved up to 90% compression on large, out-of-distribution molecules with minimal size sensitivity.
• Outperformed SPE and PeptideCLM on peptide corpora in compression and entropy metrics.
• Trie+TTG yielded more separable molecular representations and stronger QSAR predictive performance.

Conclusions:

• The Trie+TTG method provides compact, stable, and chemically meaningful tokenizations.
• This approach is suitable for modern molecular representation learning and foundation models.
• Combining trie-based mining with TTG refinement offers superior compression and generalization capabilities.