RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Chain-of-Thought (CoT) reasoning enhances large language models (LLMs) on complex tasks.
• Current limitations include a lack of quantitative metrics for measurable CoT and methods for unmeasurable CoT, such as multimodal perception.

Purpose of the Study:

• Introduce the Reasoning Boundary Framework++ (RBF++) to address the limitations in evaluating and optimizing CoT capabilities.
• Provide quantitative metrics and actionable guidelines for measurable CoT, and methods for assessing unmeasurable CoT.

Main Methods:

• Define Reasoning Boundary (RB) as the maximum CoT performance limit and propose a combination law for quantitative analysis.
• Introduce a constant assumption for unmeasurable RBs and a division mechanism to quantify domain knowledge and multimodal perception.

Main Results:

• Validated RBF++ feasibility across 38 models and 14 tasks, including cross-modal settings.
• Evaluated 10 CoT strategies, offering insights into optimization and decay.
• Expanded benchmarks for measuring RBs in LLM reasoning.

Conclusions:

• RBF++ advances the understanding and optimization of CoT reasoning in LLMs.
• The framework facilitates quantitative analysis and optimization of both measurable and unmeasurable CoT capabilities.