Beyond performance: how design choices shape chemical language models.

Area of Science:

• Computational Chemistry
• Artificial Intelligence
• Machine Learning

Background:

• Chemical language models (CLMs) excel at molecular property prediction and generation.
• Key design choices like molecular representation, tokenization, and architecture are underexplored regarding their impact on CLM interpretability.
• Understanding how these choices influence chemical information encoding is crucial for developing reliable CLMs.

Purpose of the Study:

• To systematically evaluate the impact of molecular representation, tokenization, and architecture on CLM performance and chemical interpretability.
• To investigate how different design choices affect the internal representations learned by CLMs.
• To provide guidance for developing more chemically grounded and interpretable CLMs.

Main Methods:

• Systematic evaluation of CLMs with varying design choices (representation, tokenization, architecture).
• Fine-tuning CLMs on downstream tasks to assess performance.
• Probing the latent space structure using probing predictors, vector operations, and dimensionality reduction techniques.

Main Results:

• Downstream task performance was largely similar across different model configurations.
• Substantial differences in the structure and interpretability of internal representations were observed based on design choices.
• Atomwise tokenization generally improved the chemical interpretability of CLM representations.
• A RoBERTa-based model with SMILES input proved a reliable baseline for standard prediction tasks.

Conclusions:

• Design choices significantly shape how chemical information is encoded within CLMs, impacting interpretability more than raw performance.
• Atomwise tokenization is a promising strategy for enhancing the chemical interpretability of CLMs.
• Current CLM design choices, particularly RoBERTa with SMILES, offer a solid foundation, but further research can yield more interpretable models.