Assessing the Chemical Intelligence of Large Language Models

Area of Science:

• Artificial Intelligence
• Computational Chemistry
• Organic Chemistry

Background:

• Large Language Models (LLMs) have demonstrated versatility across various domains.
• The emergence of "reasoning models" has significantly enhanced LLM capabilities in complex problem-solving areas like mathematics and software engineering.

Purpose of the Study:

• To evaluate the direct application of reasoning models to chemistry tasks without external tools.
• To introduce a novel benchmark, ChemIQ, for assessing core organic chemistry concepts and chemical reasoning.

Main Methods:

• Development of the ChemIQ benchmark with 816 short-answer questions focused on molecular comprehension and chemical reasoning.
• Assessment of reasoning models (OpenAI's o3-mini, Google's Gemini 2.5 Pro, DeepSeek R1) on the ChemIQ benchmark.
• Evaluation of model performance in converting SMILES strings to IUPAC names and elucidating structures from NMR data.

Main Results:

• Reasoning models achieved 50-57% accuracy on the ChemIQ benchmark, a substantial improvement over non-reasoning models (3-7% accuracy).
• Models demonstrated the ability to convert SMILES strings to IUPAC names and elucidate molecular structures from 1D/2D ¹H and ¹³C NMR data.
• Gemini 2.5 Pro accurately generated SMILES strings for ~90% of molecules with up to 10 heavy atoms, including one with 25 heavy atoms.

Conclusions:

• Latest reasoning models show increasing proficiency in advanced chemical reasoning tasks.
• The reasoning processes employed by these models appear to emulate those of human chemists.
• LLMs are becoming capable tools for direct application in complex chemistry problem-solving.