Assessment of chemistry knowledge in large language models that generate code

Area of Science:

• Computational Chemistry
• Artificial Intelligence in Science

Background:

• The integration of artificial intelligence, particularly large language models (LLMs), into scientific disciplines is rapidly advancing.
• Evaluating the specific domain knowledge of LLMs, such as in chemistry, is crucial for understanding their potential applications.

Purpose of the Study:

• To assess the chemistry knowledge of code-generating LLMs.
• To develop and introduce a flexible framework for evaluating LLM performance on chemistry tasks.
• To create a benchmark dataset and open-source tools for community use.

Main Methods:

• LLMs were prompted to solve chemistry problems framed as coding tasks.
• A benchmark set of chemistry problems was developed.
• Model performance was evaluated through automated code testing and expert review.
• Prompt engineering strategies were investigated to improve accuracy.

Main Results:

• Recent LLMs exhibit considerable ability to generate correct code for various chemistry topics.
• Prompt engineering techniques, such as adding copyright notices, improved LLM accuracy by up to 30 percentage points.
• The developed dataset and evaluation framework are open-source, fostering community contribution.

Conclusions:

• Code-generating LLMs possess substantial, though not perfect, knowledge of chemistry.
• LLMs show immense potential to revolutionize chemistry education and research.
• The open-source nature of the resources encourages further development and standardized evaluation of AI models in chemistry.