Tokenization for Molecular Foundation Models.

Area of Science:

• Cheminformatics and computational chemistry
• Artificial intelligence in scientific discovery
• Molecular representation and modeling

Background:

• Text-based foundation models are crucial for scientific discovery, particularly in molecular design.
• Existing molecular foundation models are limited by closed-vocabulary tokenizers, restricting their ability to represent the full molecular space.
• A systematic evaluation of 35 tokenizers revealed significant gaps in SMILES representation coverage.

Purpose of the Study:

• To systematically evaluate the coverage of existing molecular tokenizers.
• To assess the impact of tokenizer choice on molecular property prediction.
• To develop novel tokenizers with comprehensive coverage of molecular representations.

Main Methods:

• Evaluated 35 tokenizers, including 20 chemistry-specific ones, for SMILES representation coverage.
• Introduced n-gram language models as a proxy to assess tokenizer impact.
• Pretrained and finetuned 18 RoBERTa-style encoders for molecular property prediction.
• Developed two new tokenizers, Smirk and Smirk-GPE, with full OpenSMILES coverage.

Main Results:

• Identified significant gaps in the coverage of existing tokenizers for SMILES.
• Demonstrated the effectiveness of n-gram language models as a proxy for tokenizer evaluation.
• Showcased the capability of pretrained RoBERTa-style encoders for molecular property prediction.
• Introduced Smirk and Smirk-GPE tokenizers that integrate nuclear, electronic, and geometric degrees of freedom.

Conclusions:

• Existing tokenizers inadequately cover the molecular space, necessitating open-vocabulary approaches.
• The proposed Smirk and Smirk-GPE tokenizers provide full OpenSMILES coverage and enable broader applications.
• Highlights the need for chemically diverse benchmarks and open-vocabulary modeling in cheminformatics.