Measuring and mitigating debugging effectiveness decay in code language models

Area of Science:

• Artificial Intelligence
• Software Engineering
• Computer Science

Background:

• Iterative debugging is crucial for AI code generation systems.
• Current AI models exhibit significant performance degradation in debugging over successive attempts.

Purpose of the Study:

• To quantify the decay in AI debugging effectiveness.
• To introduce a framework for predicting debugging ineffectiveness.
• To propose a strategy for improving AI debugging.

Main Methods:

• Developed the Debugging Decay Index (DDI) as a mathematical framework.
• Analyzed the exponential decay pattern of AI debugging capability.
• Implemented a strategic fresh start approach to AI debugging.

Main Results:

• AI debugging effectiveness follows an exponential decay, losing 60-80% capability within 2-3 attempts.
• The DDI framework accurately predicts when AI debugging becomes ineffective.
• Strategic interventions can significantly restore AI debugging effectiveness.

Conclusions:

• Current AI self-debugging has fundamental limitations.
• The DDI provides a systematic metric for evaluating LLM-based code generation.
• A strategic fresh start approach enhances AI debugging performance.