QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry.

Area of Science:

• Quantitative chemistry
• Computational chemistry
• Artificial intelligence in chemistry

Background:

• Quantitative chemistry is crucial for modern chemical research.
• The capability of large language models (LLMs) in performing quantitative chemistry calculations is not well understood.
• Existing benchmarks do not adequately assess the mathematical reasoning skills of LLMs in chemistry.

Purpose of the Study:

• To introduce QCBench, a novel benchmark for evaluating LLMs on quantitative chemistry problems.
• To systematically assess the mathematical reasoning abilities of LLMs across various chemistry subfields.
• To identify specific computational weaknesses and limitations of LLMs in scientific problem-solving.

Main Methods:

• Developed QCBench, a benchmark with 350 computational chemistry problems across 7 subfields (analytical, bio-organic, general, inorganic, physical, polymer, and quantum chemistry).
• Categorized problems into easy, medium, and difficult tiers to systematically evaluate LLM reasoning.
• Designed problems to require explicit numerical reasoning and prevent heuristic shortcuts.
• Evaluated 24 LLMs on the QCBench dataset.

Main Results:

• LLM performance consistently degraded as the complexity of quantitative chemistry problems increased.
• Significant differences in model-specific limitations were observed across difficulty levels.
• The study highlighted a gap between the language fluency of LLMs and their accuracy in scientific computation.

Conclusions:

• QCBench provides a fine-grained diagnostic tool for assessing LLM computational weaknesses in quantitative chemistry.
• The findings underscore the need for domain-adaptive fine-tuning and multimodal integration to improve LLM performance in scientific domains.
• Future research can build upon QCBench to advance the application of LLMs in rigorous chemical calculations.