The relationship between reasoning and performance in large language models-o3 (mini) thinks harder, not longer

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Mathematical Reasoning

Background:

• Large language models (LLMs) show progress in mathematical reasoning via chain-of-thought and reinforcement learning.
• The relationship between reasoning token usage and accuracy gains in LLMs remains unclear, especially across model generations.

Purpose of the Study:

• To analyze the interplay between reasoning chain length and accuracy in LLMs.
• To determine if improved performance stems from longer reasoning or enhanced reasoning efficiency.

Main Methods:

• Systematic analysis of reasoning chain length across o1-mini and o3-mini LLM variants.
• Evaluation on the Omni-MATH benchmark, controlling for question difficulty and compute settings.

Main Results:

• o3-mini (m) achieved higher accuracy than o1-mini without increasing reasoning length.
• Accuracy generally decreased as reasoning chains lengthened across all models.
• More proficient models exhibited a smaller accuracy drop with longer chains, indicating more effective test-time compute usage.
• o3-mini (h) showed a marginal accuracy gain over o3-mini (m) but used significantly more reasoning tokens.

Conclusions:

• LLM advancements improve reasoning efficiency rather than solely relying on longer chains.
• Increased reasoning length can decrease accuracy, a phenomenon better managed by newer, more capable models.
• Future LLM development and evaluation should consider reasoning efficiency and compute effectiveness.