MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data.

Area of Science:

• Artificial Intelligence
• Mathematics Education

Background:

• Large language models (LLMs) excel at natural language tasks but struggle with complex mathematical reasoning.
• Evaluating LLM mathematical capabilities requires specialized datasets for rigorous assessment.

Purpose of the Study:

• Introduce MathOdyssey, a novel dataset for evaluating mathematical reasoning in LLMs.
• Provide a standardized resource for reproducible LLM performance assessment in mathematics.

Main Methods:

• Curated 387 expert-generated mathematical problems from high school to Olympiad levels.
• Included detailed solutions and categorized problems by difficulty, subject, and answer type.
• Developed dataset through expert contributions, peer review, and standardized formatting.

Main Results:

• Evaluated performance of representative LLMs on the MathOdyssey dataset.
• Reported LLM performance across various problem types and difficulty levels.

Conclusions:

• MathOdyssey serves as an open-access resource for fine-grained assessment of LLM mathematical abilities.
• The dataset will foster research in AI-driven mathematical reasoning and education.