On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Aligning large language models (LLMs) with human preferences is vital for decision-making.
• Current reinforcement learning from human feedback (RLHF) methods may introduce algorithmic bias, potentially ignoring minority preferences (preference collapse).

Purpose of the Study:

• To introduce a novel approach, preference matching (PM) RLHF, that mitigates algorithmic bias in LLM alignment.
• To provably align LLMs with the reward model's preference distribution using the Bradley-Terry-Luce/Plackett-Luce model.

Main Methods:

• Developed a PM regularizer (negative logarithm of the LLM's policy probability distribution) to balance response diversification and reward maximization.
• Derived the PM regularizer by solving an ordinary differential equation.
• Introduced a conditional variant of PM RLHF for natural language generation.

Main Results:

• Conditional PM RLHF demonstrated significant improvements in alignment with human preferences.
• Experiments on OPT and Llama-family models showed a 29% to 41% improvement compared to standard RLHF.

Conclusions:

• Preference matching RLHF offers a provably effective method for mitigating algorithmic bias in LLM alignment.
• The conditional PM RLHF approach enhances fairness and accuracy in aligning LLMs with human preferences, particularly for natural language generation tasks.