DeepSeek-R1は,強化学習を通じてLLMにおける推論を奨励する
Daya Guo1, Dejian Yang1, Haowei Zhang1
1DeepSeek-AI Team, Hangzhou, China.
Nature
|September 17, 2025
まとめ
補強学習 (RL) は,人間のデータなしで大きな言語モデル (LLM) の推論を強化します. このアプローチは,複雑なタスクのパフォーマンスを改善するために,高度なAI推論パターンを育成します.
科学分野:
- 人工知能
- 機械学習
背景:
- 一般的な推論は AIの核心課題です
- 大規模な言語モデル (LLM) と思考の連鎖 (CoT) の誘導は有望ですが,広範な人間データが必要です.
- 現在のLLM機能は 複雑な推論のタスクには不十分です
研究 の 目的:
- 純粋な補強学習 (RL) がLLMの推論能力を向上させることを示す.
- 人間による推論の軌跡の必要性を回避するためです
- LLMにおける高度な推論パターンの発展を促進する.
主な方法:
- LLMのための純粋な強化学習 (RL) フレームワークの実施
- 発覚した推論パターンを刺激するためにRLを使用するLLMのトレーニング.
- 検証可能なタスクに関するRLで訓練されたLLMのパフォーマンスを評価する.
主要な成果:
- RLフレームワークは 自己反省や検証などの 発現した推論パターンを促進しました
- RLで訓練されたLLMは,数学,コーディング,STEMのタスクで監督学習の同級生を上回りました.
- 大型モデルから生まれる推論パターンは 小型モデルの能力を高めることができます
結論:
- 純粋な補強学習は,人間の実証なしにLLMの推論を効果的に強化します.
- RLで訓練されたLLMは,複雑で検証可能なタスクで優れたパフォーマンスを発揮します.
- 開発された RL フレームワークは,AI 推論を進めるためのスケーラブルな方法を提供し,より小さなモデルを導くことができます.
関連する概念動画
Reinforcement
841
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
841
Observational Learning
841
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
841
Reinforcement Schedules
460
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
460
Reasoning
400
Reasoning is the action of thinking about something in a logical, sensible way. It is integral to problem-solving, decision-making, and critical thinking. Reasoning can be inductive or deductive. Reasoning involves transforming information into conclusions, which is essential for problem-solving, decision-making, and critical thinking.
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
400
Purposive Learning
447
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
447
Inductive Reasoning
65.2K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
65.2K


