NMR-Challenge for LLMs: Evaluating Chemical Reasoning in Humans and AI.

Area of Science:

• Chemistry
• Artificial Intelligence
• Computational Science

Background:

• Nuclear Magnetic Resonance (NMR) structure determination is a complex task requiring significant expertise.
• Solving NMR spectra involves critical evaluation of spectral data features.
• Large Language Models (LLMs) and Large Reasoning Models (LRMs) show potential for automating chemical analysis.

Purpose of the Study:

• To evaluate the capabilities of various LLMs and LRMs in solving NMR spectral structure elucidation problems.
• To compare the performance of LLMs/LRMs against undergraduate organic chemistry students.
• To benchmark LLM reasoning abilities on complex chemistry tasks.

Main Methods:

• 112 NMR structure elucidation problems from NMR-Challenge.com were used.
• A plain text format was developed to evaluate LLM reasoning.
• Ten LLMs were tested with five different prompting strategies, including providing background chemistry knowledge and reasoning strategies.
• Performance was compared to undergraduate students and historical human submissions.

Main Results:

• Newer LLMs trained for reasoning performed better.
• Advanced reasoning models (e.g., o1) exceeded undergraduate student performance across Easy, Moderate, and Hard problem sets.
• LLM errors in NMR structure elucidation mirrored common human mistakes but also showed unique patterns.
• Prompting strategies and temperature variations had minimal impact on performance.

Conclusions:

• LLMs, particularly advanced reasoning models, can now exceed undergraduate student performance in NMR structure elucidation.
• LLMs offer a valuable tool for benchmarking reasoning in chemistry.
• While LLMs mimic some human errors, their distinct reasoning processes warrant further investigation.