通过强化学习激励LLM中的推理
Daya Guo1, Dejian Yang1, Haowei Zhang1
1DeepSeek-AI Team, Hangzhou, China.
Nature
|September 17, 2025
概括
强化学习 (RL) 增强了大型语言模型 (LLM) 在没有人类数据的情况下的推理. 这种方法促进了先进的AI推理模式,以提高复杂任务的性能.
科学领域:
- 人工智能
- 机器学习
背景情况:
- 一般推理是人工智能的核心挑战.
- 大型语言模型 (LLM) 和思维链 (CoT) 提示是有希望的,但需要大量的人类数据.
- 目前的LLM能力不足以完成复杂的推理任务.
研究的目的:
- 证明纯粹的强化学习 (RL) 可以增强LLM的推理能力.
- 避免需要人为注释的推理轨迹.
- 促进LLM中先进的推理模式的新兴发展.
主要方法:
- 实施纯强化学习 (RL) 框架.
- 培训使用RL的LLM来激励新出现的推理模式.
- 评估经过RL培训的LLM在可验证任务中的表现.
主要成果:
- 理性学习框架促进了自我反思和验证等新出现的推理模式.
- 在数学,编码和STEM任务上超越了受监督的学习对手.
- 来自大型模型的新兴推理模式可以增强小型模型的功能.
结论:
- 纯粹的强化学习有效地增强了LLM的推理,而不需要人类演示.
- 经过RL培训的LLM在复杂,可验证的任务中表现出卓越的表现.
- 开发的RL框架为推进人工智能推理提供了可扩展的方法,并可以指导较小的模型.
相关概念视频
Reinforcement
841
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
841
Observational Learning
841
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
841
Reinforcement Schedules
460
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
460
Reasoning
400
Reasoning is the action of thinking about something in a logical, sensible way. It is integral to problem-solving, decision-making, and critical thinking. Reasoning can be inductive or deductive. Reasoning involves transforming information into conclusions, which is essential for problem-solving, decision-making, and critical thinking.
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
400
Purposive Learning
447
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
447
Inductive Reasoning
65.2K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
65.2K


