Evaluating large language models on multimodal chemistry olympiad exams

Area of Science:

• Artificial Intelligence
• Chemistry Education

Background:

• Multimodal large language models (LLMs) face challenges in scientific reasoning, especially in chemistry, which involves complex visual and symbolic data.
• Effective integration of visual and textual information is crucial for advanced problem-solving in chemistry.

Purpose of the Study:

• To systematically evaluate the performance of 40 multimodal LLMs on chemistry-specific reasoning tasks.
• To identify limitations in current models' ability to fuse information from diverse modalities.
• To assess the impact of Chain-of-Thought prompting on multimodal scientific reasoning.

Main Methods:

• Curated benchmark of U.S. National Chemistry Olympiad (USNCO) questions spanning over two decades.
• Systematic evaluation of 40 proprietary and open-source multimodal LLMs.
• Ablation studies and occlusion-based interpretability to analyze model behavior and visual grounding.
• Comparison of performance with and without Chain-of-Thought prompting.

Main Results:

• Many evaluated multimodal LLMs exhibit significant difficulties with modality fusion, sometimes performing worse with visual input.
• Chain-of-Thought prompting demonstrably improves both accuracy and visual grounding in multimodal chemistry problem-solving.
• Performance variations highlight critical limitations in current multimodal LLMs' scientific reasoning capabilities.

Conclusions:

• Current multimodal LLMs require substantial improvements for robust scientific reasoning in chemistry.
• Chain-of-Thought prompting offers a promising strategy for enhancing multimodal AI performance and interpretability.
• The developed benchmark serves as a critical tool for measuring progress in domain-specific multimodal AI research.