Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) rely on preference alignment for success.
• Preference alignment often results in poor model calibration, a phenomenon known as overconfidence.
• Pre-trained models are typically well-calibrated, but LLMs degrade after alignment.

Purpose of the Study:

• Investigate the reasons behind calibration degradation in LLMs post-preference alignment.
• Develop methods to address and mitigate poor calibration in aligned LLMs.
• Analyze the impact of calibration on LLM performance and propose solutions for different model regimes.

Main Methods:

• Observed that preference collapse during alignment generalizes to calibration issues, causing overconfidence.
• Demonstrated the effectiveness of fine-tuning with domain-specific knowledge to reduce overconfidence.
• Categorized models into 'calibratable' and 'non-calibratable' based on Expected Calibration Error (ECE).
• Proposed a calibration-aware fine-tuning approach for the calibratable regime.
• Developed an EM-algorithm-based ECE regularization for the non-calibratable regime.

Main Results:

• Preference alignment leads to overconfidence and poor calibration in LLMs.
• Domain-specific fine-tuning alleviates overconfidence.
• A calibration-aware fine-tuning approach maintains performance in the calibratable regime.
• ECE regularization effectively reduces calibration error in the non-calibratable regime.
• Proposed methods were validated through extensive experiments.

Conclusions:

• Preference alignment negatively impacts LLM calibration due to preference collapse.
• Domain-specific knowledge and calibration-aware fine-tuning are crucial for improving LLM calibration.
• Tailored methods for calibratable and non-calibratable models effectively address overconfidence and maintain performance.