在LLM培训中使用基于共识的奖励来缓解恶意RLHF反的框架
Zafaryab Haider1, Md Hafizur Rahman2, Vijay Devabhaktuni3
1Department of Electrical and Computer Engineering (ECE), University of Maine, Orono, ME, USA. zafaryab.haider@maine.edu.
Scientific reports
|March 18, 2025
概括
一个名为COBRA的新框架解决了使用人类反 (RLHF) 强化学习训练大型语言模型 (LLM) 的安全风险. 科布拉有效地过出恶意的人类反,提高了LLM在现实应用中的性能和安全性.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 在各个行业越来越多地被采用,但面临着安全和隐私方面的挑战.
- 强化学习从人类反 (RLHF) 对于LLM培训至关重要,传授人类的品质.
- RLHF过程容易受到恶意反的影响,可能会降低LLM的性能并导致有害的输出.
研究的目的:
- 提出一个新的框架,COBRA (基于共识的奖励),以减轻RLHF的恶意反.
- 在混合信任环境中增强LLM培训性能和稳定性.
- 为了验证COBRA对最先进的方法的有效性.
主要方法:
- 开发了COBRA框架,这是一个基于共识的技术,用于在RLHF期间过噪音的人类反.
- 通过使用各种LLM模型 (例如,GPT-2 XL) 评估了COBRA对情绪分析和对话任务的用例.
- 将COBRA的性能与标准RLHF和先前的方法进行比较 (Coste等. ) 的情况.
主要成果:
- COBRA显著提高了LLM的性能,通过[公式:查看文本]进行对话任务和[公式:查看文本]进行情绪分析来优于不受保护的奖励生成.
- 量化比较显示,COBRA实现了最先进的性能,特别是使用更少的奖励模型.
- 在较少的奖励模型中,COBRA证明了奖励准确度的提高 ([公式:见文本]).
结论:
- COBRA有效地中和了RLHF中的恶意反,提高了LLM培训结果.
- 拟议的框架为在关键应用中安全可靠的LLM开发提供了强大的解决方案.
- 科巴在确保法学士培训数据的完整性和质量方面取得了重大进展.
相关概念视频
01:24Effects of feedback
496
Feedback in control systems plays a critical role in shaping various operational parameters, extending beyond simple error reduction to influence stability, bandwidth, gain, impedance, and sensitivity. Understanding these effects requires examining a basic feedback system characterized by defined input, output, error, and feedback signals.
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
496
01:28Load-frequency control
106
Load-frequency control (LFC) is vital for maintaining power system stability, ensuring that frequency and power flows remain within acceptable limits during load changes. Turbine-governor control eliminates rotor accelerations and decelerations following load changes. However, a steady-state frequency error persists when the change in the turbine-governor reference setting is zero. In an interconnected power system, each area agrees to export or import a scheduled amount of power through...
106
01:23Reinforcement
169
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
169
01:26Feedback control systems
262
Feedback control systems are categorized in various ways based on their design, analysis, and signal types.
Linear feedback systems are theoretical models that simplify analysis and design. These systems operate under the principle that their output is directly proportional to their input within certain ranges. For instance, an amplifier in a control system behaves linearly as long as the input signal remains within a specific range. However, most physical systems exhibit inherent nonlinearity...
Linear feedback systems are theoretical models that simplify analysis and design. These systems operate under the principle that their output is directly proportional to their input within certain ranges. For instance, an amplifier in a control system behaves linearly as long as the input signal remains within a specific range. However, most physical systems exhibit inherent nonlinearity...
262
01:31Confirmation Biases
5.4K
The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
5.4K
01:06Law of Effect
1.3K
B.F. Skinner, a prominent figure in behavioral psychology, introduced operant conditioning by emphasizing the role of consequences in shaping behavior. This theory builds upon the law of effect proposed by Edward Thorndike, which posits that behaviors followed by satisfying outcomes are likely to be repeated. In contrast, those followed by unsatisfying outcomes are less likely to recur.
Edward Thorndike's foundational work involved studying learning in animals, particularly using puzzle...
Edward Thorndike's foundational work involved studying learning in animals, particularly using puzzle...
1.3K