A framework for mitigating malicious RLHF feedback in LLM training using consensus based reward.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) are increasingly adopted across industries, but face security and privacy challenges.
• Reinforcement Learning from Human Feedback (RLHF) is crucial for LLM training, imparting human-like qualities.
• The RLHF process is vulnerable to malicious feedback, potentially degrading LLM performance and causing harmful outputs.

Purpose of the Study:

• To propose a novel framework, COBRA (COnsensus-Based RewArd), to mitigate malicious feedback in RLHF.
• To enhance LLM training performance and robustness in mixed-trust environments.
• To validate COBRA's effectiveness against state-of-the-art methods.

Main Methods:

• Developed the COBRA framework, a consensus-based technique for filtering noisy human feedback during RLHF.
• Evaluated COBRA on Sentiment Analysis and Conversational Task use cases using various LLM models (e.g., GPT-2 XL).
• Compared COBRA's performance against standard RLHF and a prior method (Coste et al.).

Main Results:

• COBRA significantly improved LLM performance, outperforming unprotected reward generation by [Formula: see text] for conversational tasks and [Formula: see text] for sentiment analysis.
• Quantitative comparisons showed COBRA achieved state-of-the-art performance, especially with fewer reward models.
• COBRA demonstrated increased reward accuracy ([Formula: see text]) at a lower number of reward models ([Formula: see text]).

Conclusions:

• COBRA effectively neutralizes malicious feedback in RLHF, enhancing LLM training outcomes.
• The proposed framework offers a robust solution for secure and reliable LLM development in critical applications.
• COBRA presents a significant advancement in ensuring the integrity and quality of LLM training data.