Measuring Chemical LLM robustness to molecular representations: a SMILES variation-based framework

Area of Science:

• Computational Chemistry
• Cheminformatics
• Drug Discovery

Background:

• Natural Language Processing (NLP) integration advances drug discovery.
• Molecule representations in language models (LMs) are key for chemical understanding.

Purpose of the Study:

• To explore LM ability to match chemical structures across different representations.
• To develop a framework for assessing the robustness of chemistry LMs.

Main Methods:

• Proposed Augmented Molecular Retrieval (AMORE), a zero-shot framework.
• Utilized SMILES augmentations to maintain foundational chemical structures.
• Facilitated similarity between embedding representations of molecules and their variations.

Main Results:

• Tested chemistry LMs (ChemLLMs) on molecular captioning (ChEBI-20) and MoleculeNet benchmarks.
• Demonstrated that current ChemLLMs are not robust to different SMILES representations.
• Observed that SMILES string variations impact model results, aligning with the AMORE framework.

Conclusions:

• Current chemistry LMs lack robustness in handling diverse molecular representations.
• The AMORE framework effectively assesses LM performance on varied chemical inputs.
• Further development is needed to enhance LM understanding of chemical structures for drug discovery.