995 lines
82 KiB
Plaintext
995 lines
82 KiB
Plaintext
TITLE: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
|
||
ARXIV: 2501.12948
|
||
AUTHORS: DeepSeek-AI , Daya Guo , Dejian Yang , Haowei Zhang , Junxiao Song , Peiyi Wang , Qihao Zhu , Runxin Xu , Ruoyu Zhang , Shirong Ma , Xiao Bi , Xiaokang Zhang , Xingkai Yu , Yu Wu , Z.F. Wu , Zhibin Gou , Zhihong Shao , Zhuoshu Li , Ziyi Gao , Aixin Liu , Bing Xue , Bingxuan Wang , Bochao Wu , Bei Feng , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenyu Zhang , Chong Ruan , Damai Dai , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fucong Dai , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Han Bao , Hanwei Xu , Haocheng Wang , Honghui Ding , Huajian Xin , Huazuo Gao , Hui Qu , Hui Li , Jianzhong Guo , Jiashi Li , Jiawei Wang , Jingchang Chen , Jingyang Yuan , Junjie Qiu , Junlong Li , J.L. Cai , Jiaqi Ni , Jian Liang , Jin Chen , Kai Dong , Kai Hu , Kaige Gao , Kang Guan , Kexin Huang , Kuai Yu , Lean Wang , Lecong Zhang , Liang Zhao , Litong Wang , Liyue Zhang , Lei Xu , Leyi Xia , Mingchuan Zhang , Minghua Zhang , Minghui Tang , Meng Li , Miaojun Wang , Mingming Li , Ning Tian , Panpan Huang , Peng Zhang , Qiancheng Wang , Qinyu Chen , Qiushi Du , Ruiqi Ge , Ruisong Zhang , Ruizhe Pan , Runji Wang , R.J. Chen , R.L. Jin , Ruyi Chen , Shanghao Lu , Shangyan Zhou , Shanhuang Chen , Shengfeng Ye , Shiyu Wang , Shuiping Yu , Shunfeng Zhou , Shuting Pan , S.S. Li , Shuang Zhou , Shaoqing Wu , Shengfeng Ye , Tao Yun , Tian Pei , Tianyu Sun , T. Wang , Wangding Zeng , Wanjia Zhao , Wen Liu , Wenfeng Liang , Wenjun Gao , Wenqin Yu , Wentao Zhang , W.L. Xiao ,
|
||
DATE: [Submitted on 22 Jan 2025 ( v1 ), last revised 4 Jan 2026 (this version, v2)]
|
||
|
||
=== SECTION HEADINGS ===
|
||
## Report GitHub Issue
|
||
## DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
|
||
## Abstract
|
||
## keywords
|
||
## 1 Introduction
|
||
## 2 DeepSeek-R1-Zero
|
||
## 2.1 Group Relative Policy Optimization
|
||
## 2.2 Reward Design
|
||
## 2.3 Incentivize Reasoning Capability in LLMs
|
||
## 3 DeepSeek-R1
|
||
## 3.1 Model-based Rewards
|
||
## Helpful Reward Model
|
||
## Safety Reward Model
|
||
## 3.2 Training Details
|
||
## 3.2.1 Training Details of the First RL Stage
|
||
## 3.2.2 Training Details of the Second RL Stage
|
||
## 4 Experiment
|
||
## 5 Ethics and Safety Statement
|
||
## 6 Conclusion, Limitation, and Future Work
|
||
## 7 Author List
|
||
## Appendix
|
||
## Appendix A Background
|
||
## A.1 DeepSeek-V3
|
||
## A.2 Conventional Post-Training Paradigm
|
||
## A.3 A Comparison of GRPO and PPO
|
||
## Appendix B Training Details
|
||
## B.1 RL Infrastructure
|
||
## B.2 Reward Model Prompt
|
||
## B.3 Data Recipe
|
||
## B.3.1 RL Data
|
||
## B.3.2 DeepSeek-R1 Cold Start
|
||
## Question
|
||
## Thought process
|
||
## Question
|
||
## Response
|
||
## Question
|
||
## Response
|
||
## Question
|
||
## Response
|
||
## Answer Quality Classification
|
||
## Question
|
||
## Reference Answer
|
||
## Answer to be Evaluated
|
||
## Output Format
|
||
## B.3.3 800K Supervised Data
|
||
## Reasoning Data
|
||
## Non-Reasoning Data
|
||
## SFT Data Statistics
|
||
## Examples of SFT Trajectories
|
||
## Question
|
||
## Response
|
||
## Question
|
||
## Response
|
||
## Question
|
||
## Response
|
||
## B.4 Hyper-Parameters
|
||
## B.4.1 Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B
|
||
## B.4.2 Hyper-Parameters of SFT
|
||
## B.4.3 Hyper-Parameters of Distillation
|
||
## B.4.4 Training Cost
|
||
## B.5 Reward Hacking
|
||
## B.6 Ablation Study of Language Consistency Reward
|
||
## Appendix C Self-Evolution of DeepSeek-R1-Zero
|
||
## C.1 Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training
|
||
## C.2 Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training
|
||
## Appendix D Evaluation of DeepSeek-R1
|
||
## D.1 Experiment Setup
|
||
## Benchmarks
|
||
## Decontamination
|
||
## Evaluation Prompts
|
||
## Baselines
|
||
## D.2 Main Results
|
||
## Standard Benchmark
|
||
## Human Evaluation
|
||
## D.3 DeepSeek-R1 Safety Report
|
||
## D.3.1 Risk Control System for DeepSeek-R1
|
||
## Potential Risky Dialogue Filtering
|
||
## Model-based Risk Review
|
||
## D.3.2 R1 Safety Evaluation on Standard Benchmarks
|
||
## D.3.3 Safety Taxonomic Study of R1 on In-House Benchmark
|
||
## Discrimination and Prejudice Issues
|
||
## Illegal and Criminal Behavior
|
||
## Harmful Behavior
|
||
## Moral and Ethical Issues
|
||
## D.3.4 Multilingual Safety Performance
|
||
## D.3.5 Robustness against Jailbreaking
|
||
## Appendix E More Analysis
|
||
## E.1 Performance Comparison with DeepSeek-V3
|
||
## E.2 Generalization to Real-World Competitions
|
||
## E.3 Mathematical Capabilities Breakdown by Categories
|
||
## E.4 An Analysis on CoT Length
|
||
## E.5 Performance of Each Stage on Problems of Varying Difficulty
|
||
## Appendix F DeepSeek-R1 Distillation
|
||
## F.1 Distillation v.s. Reinforcement Learning
|
||
## Appendix G Discussion
|
||
## G.1 Key Findings
|
||
## G.2 Unsuccessful Attempts
|
||
## Process Reward Model (PRM)
|
||
## Monte Carlo Tree Search (MCTS)
|
||
## Appendix H Related Work
|
||
## H.1 Chain-of-thought Reasoning
|
||
## H.2 Scaling Inference-time Compute
|
||
## H.3 Reinforcement Learning for Reasoning Enhancement
|
||
## Appendix I Open Weights, Code, and Data
|
||
## Appendix J Evaluation Prompts and Settings
|
||
## Question:
|
||
## Choices:
|
||
## Instruction
|
||
## Instruction
|
||
## Model Outputs
|
||
## Task
|
||
## Best Model Identifier
|
||
|
||
=== BODY (head) ===
|
||
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
|
||
## Report GitHub Issue
|
||
×
|
||
Title:
|
||
Content selection saved. Describe the issue below:
|
||
Description:
|
||
Submit without GitHub
|
||
Submit in GitHub
|
||
arXiv is now an independent nonprofit!
|
||
Learn more
|
||
×
|
||
Back to arXiv
|
||
Why HTML?
|
||
Report Issue
|
||
Back to Abstract
|
||
Download PDF
|
||
Abstract
|
||
1 Introduction
|
||
2 DeepSeek-R1-Zero
|
||
2.1 Group Relative Policy Optimization
|
||
2.2 Reward Design
|
||
2.3 Incentivize Reasoning Capability in LLMs
|
||
3 DeepSeek-R1
|
||
3.1 Model-based Rewards
|
||
Helpful Reward Model
|
||
Safety Reward Model
|
||
3.2 Training Details
|
||
3.2.1 Training Details of the First RL Stage
|
||
3.2.2 Training Details of the Second RL Stage
|
||
4 Experiment
|
||
5 Ethics and Safety Statement
|
||
6 Conclusion, Limitation, and Future Work
|
||
7 Author List
|
||
A Background
|
||
A.1 DeepSeek-V3
|
||
A.2 Conventional Post-Training Paradigm
|
||
A.3 A Comparison of GRPO and PPO
|
||
B Training Details
|
||
B.1 RL Infrastructure
|
||
B.2 Reward Model Prompt
|
||
B.3 Data Recipe
|
||
B.3.1 RL Data
|
||
B.3.2 DeepSeek-R1 Cold Start
|
||
B.3.3 800K Supervised Data
|
||
Reasoning Data
|
||
Non-Reasoning Data
|
||
SFT Data Statistics
|
||
Examples of SFT Trajectories
|
||
B.4 Hyper-Parameters
|
||
B.4.1 Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B
|
||
B.4.2 Hyper-Parameters of SFT
|
||
B.4.3 Hyper-Parameters of Distillation
|
||
B.4.4 Training Cost
|
||
B.5 Reward Hacking
|
||
B.6 Ablation Study of Language Consistency Reward
|
||
C Self-Evolution of DeepSeek-R1-Zero
|
||
C.1 Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training
|
||
C.2 Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training
|
||
D Evaluation of DeepSeek-R1
|
||
D.1 Experiment Setup
|
||
Benchmarks
|
||
Decontamination
|
||
Evaluation Prompts
|
||
Baselines
|
||
D.2 Main Results
|
||
Standard Benchmark
|
||
Human Evaluation
|
||
D.3 DeepSeek-R1 Safety Report
|
||
D.3.1 Risk Control System for DeepSeek-R1
|
||
Potential Risky Dialogue Filtering
|
||
Model-based Risk Review
|
||
D.3.2 R1 Safety Evaluation on Standard Benchmarks
|
||
D.3.3 Safety Taxonomic Study of R1 on In-House Benchmark
|
||
Discrimination and Prejudice Issues
|
||
Illegal and Criminal Behavior
|
||
Harmful Behavior
|
||
Moral and Ethical Issues
|
||
D.3.4 Multilingual Safety Performance
|
||
D.3.5 Robustness against Jailbreaking
|
||
E More Analysis
|
||
E.1 Performance Comparison with DeepSeek-V3
|
||
E.2 Generalization to Real-World Competitions
|
||
E.3 Mathematical Capabilities Breakdown by Categories
|
||
E.4 An Analysis on CoT Length
|
||
E.5 Performance of Each Stage on Problems of Varying Difficulty
|
||
F DeepSeek-R1 Distillation
|
||
F.1 Distillation v.s. Reinforcement Learning
|
||
G Discussion
|
||
G.1 Key Findings
|
||
G.2 Unsuccessful Attempts
|
||
Process Reward Model (PRM)
|
||
Monte Carlo Tree Search (MCTS)
|
||
H Related Work
|
||
H.1 Chain-of-thought Reasoning
|
||
H.2 Scaling Inference-time Compute
|
||
H.3 Reinforcement Learning for Reasoning Enhancement
|
||
I Open Weights, Code, and Data
|
||
J Evaluation Prompts and Settings
|
||
References
|
||
License: arXiv.org perpetual non-exclusive license
|
||
arXiv:2501.12948v2 [cs.CL] 04 Jan 2026
|
||
\reportnumber
|
||
001
|
||
## DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
|
||
DeepSeek-AI
|
||
research@deepseek.com
|
||
## Abstract
|
||
General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) ( Brown et al., 2020 ; OpenAI, 2023 ) and chain-of-thought prompting ( Wei et al., 2022b ) , have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models’ capabilities are still insufficient for more complex problems.
|
||
Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.
|
||
## keywords
|
||
Large Language Model, Reasoning, Reinforcement Learning
|
||
## 1 Introduction
|
||
Reasoning capability, the cornerstone of human intelligence, enables complex cognitive tasks ranging from mathematical problem-solving to logical deduction and programming. Recent advances in artificial intelligence have demonstrated that large language models (LLMs) can exhibit emergent behaviors, including reasoning abilities, when scaled to a sufficient size ( Wei et al., 2022a ; Kaplan et al., 2020 ) . However, achieving such capabilities in pre-training typically demands substantial computational resources.
|
||
In parallel, a complementary line of research has demonstrated that large language models can be effectively augmented through chain-of-thought (CoT) prompting. This technique, which involves either providing carefully designed few-shot examples or using minimalistic prompts such as “Let’s think step by step” ( Wei et al., 2022b ; Kojima et al., 2022 ) , enables models to produce intermediate reasoning steps, thereby substantially enhancing their performance on complex tasks.
|
||
Similarly, further performance gains have been observed when models learn high-quality, multi-step reasoning trajectories during the post-training phase ( OpenAI, 2023 ; Chung et al., 2024 ) .
|
||
Despite their effectiveness, these approaches exhibit notable limitations. Their dependence on human-annotated reasoning traces hinders scalability and introduces cognitive biases. Furthermore, by constraining models to replicate human thought processes, their performance is inherently capped by the human-provided exemplars, which prevents the exploration of superior, non-human-like reasoning pathways.
|
||
To tackle these issues, we aim to explore the potential of LLMs for developing reasoning abilities through self-evolution in an RL framework, with minimal reliance on human labeling efforts.
|
||
Specifically, we build upon DeepSeek-V3-Base ( DeepSeek-AI, 2024b ) and employ Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) as our RL framework. The reward signal is solely based on the correctness of final predictions against ground-truth answers, without imposing constraints on the reasoning process itself. Notably, we bypass the conventional supervised fine-tuning (SFT) phase before RL training. This design choice stems from our hypothesis that human-defined reasoning patterns may limit model exploration, whereas unrestricted RL training can better incentivize the emergence of novel reasoning capabilities in LLMs.
|
||
Through this process, detailed in Section 2 , our model (referred to as DeepSeek-R1-Zero) naturally developed diverse and sophisticated reasoning behaviors.
|
||
In solving reasoning problems, the model exhibits a tendency to generate longer responses, incorporating verification, reflection, and the exploration of alternative approaches within each response. Although we do not explicitly teach the model how to reason, it successfully learns improved reasoning strategies through reinforcement learning.
|
||
Although DeepSeek-R1-Zero demonstrates excellent reasoning capabilities, it faces challenges such as poor readability and language mixing, occasionally combining English and Chinese within a single chain-of-thought response. Furthermore, the rule-based RL training stage of DeepSeek-R1-Zero is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering.
|
||
To address these challenges, we introduce DeepSeek-R1, a model trained through a multi-stage learning framework that integrates rejection sampling, reinforcement learning, and supervised fine-tuning, detailed in Section 3 . This training pipeline enables DeepSeek-R1 to inherit the reasoning capabilities of its predecessor, DeepSeek-R1-Zero, while aligning model behavior with human preferences through additional non-reasoning data.
|
||
To enable broader access to
|
||
powerful AI at a lower energy cost, we have distilled several smaller models and made them publicly available. These distilled models exhibit strong reasoning capabilities, surpassing the performance of their original instruction-tuned counterparts. We believe that these instruction-tuned versions will also significantly contribute to the research community by providing a valuable resource for understanding the mechanisms underlying long chain-of-thought (CoT) reasoning models and for fostering the development of more powerful reasoning models. We release DeepSeek-R1 series models to the public at https://huggingface.co/deepseek-ai .
|
||
## 2 DeepSeek-R1-Zero
|
||
We begin by elaborating on the training of DeepSeek-R1-Zero, which relies exclusively on reinforcement learning without supervised fine-tuning. To facilitate large-scale RL efficiency, we adopt Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) .
|
||
## 2.1 Group Relative Policy Optimization
|
||
GRPO ( Shao et al., 2024 ) is the reinforcement learning algorithm that we adopt to train DeepSeek-R1-Zero and DeepSeek-R1. It was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , which is widely used in the RL stage of LLMs ( Ouyang et al., 2022 ) .
|
||
For each question q q , GRPO samples a group of outputs { o 1 , o 2 , ⋯ , o G } \{o_{1},o_{2},\cdots,o_{G}\} from the old policy π θ o l d \pi_{\theta_{old}} and then optimizes the policy model π θ \pi_{\theta} by maximizing the following objective:
|
||
𝒥 G R P O ( θ ) = 𝔼 [ q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ o l d ( O | q ) ] 1 G ∑ i = 1 G ( min ( π θ ( o i | q ) π θ o l d ( o i | q ) A i , clip ( π θ ( o i | q ) π θ o l d ( o i | q ) , 1 − ε , 1 + ε ) A i ) − β 𝔻 K L ( π θ | | π r e f ) ) , \begin{split}\footnotesize&\mathcal{J}_{GRPO}(\theta)=\mathbb{E}{[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\
|
||
&\frac{1}{G}\sum_{i=1}^{G}\left(\min\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\text{clip}\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)},1-\varepsilon,1+\varepsilon\right)A_{i}\right)-\beta\mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)\right),\end{split}
|
||
(1)
|
||
𝔻 K L ( π θ | | π r e f ) = π r e f ( o i | q ) π θ ( o i | q ) − log π r e f ( o i | q ) π θ ( o i | q ) − 1 , \mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)=\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1,
|
||
(2)
|
||
where π r e f \pi_{ref} is a reference policy, ε \varepsilon and β \beta are hyper-parameters, and A i A_{i} is the advantage, computed using a group of rewards { r 1 , r 2 , … , r G } \{r_{1},r_{2},\ldots,r_{G}\} corresponding to the outputs within each group:
|
||
A i = r i − m e a n ( { r 1 , r 2 , ⋯ , r G } ) s t d ( { r 1 , r 2 , ⋯ , r G } ) . A_{i}=\frac{r_{i}-{\mathrm{m}ean(\{r_{1},r_{2},\cdots,r_{G}\})}}{{\mathrm{s}td(\{r_{1},r_{2},\cdots,r_{G}\})}}.
|
||
(3)
|
||
We give a comparison of GRPO and PPO in Supplementary A.3 .
|
||
To train DeepSeek-R1-Zero, we set the learning rate to 3e-6, the KL coefficient to 0.001, and the sampling temperature to 1 for rollout. For each question, we sample 16 outputs with a maximum length of 32,768 tokens before the 8.2k step and 65,536 tokens afterward. As a result, both the performance and response length of DeepSeek-R1-Zero exhibit a significant jump at the 8.2k step, with training continuing for a total of 10,400 steps, corresponding to 1.6 training epochs. Each training step consists of 32 unique questions, resulting in a training batch size of 512. Every 400 steps, we replace the reference model with the latest policy model. To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 mini-batches and trained for only a single inner epoch.
|
||
Table 1: Template for DeepSeek-R1-Zero. prompt will be replaced with the specific reasoning question during training.
|
||
A conversation between User and Assistant. The user asks a question, and the Assistant solves it.
|
||
The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
|
||
The reasoning process and answer are enclosed within <think>...</think> and <answer>...</answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer> .
|
||
User: prompt . Assistant:
|
||
Our high-performance RL infrastructure is described in Supplementary B.1 , ensuring scalable and efficient training.
|
||
## 2.2 Reward Design
|
||
The reward is the source of the training signal, which decides the direction of RL optimization.
|
||
For DeepSeek-R1-Zero, we employ rule-based rewards to
|
||
deliver precise feedback for data in mathematical, coding, and logical reasoning domains. Our rule-based reward system mainly consists of two types of rewards: accuracy rewards and format rewards.
|
||
Accuracy rewards evaluate whether the response is correct. For example, in the case of math problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification of correctness. Similarly, for code competition prompts, a compiler can be utilized to evaluate the model’s responses against a suite of predefined test cases, thereby generating objective feedback on correctness.
|
||
Format rewards complement the accuracy reward model by enforcing specific formatting requirements. In particular, the model is incentivized to encapsulate its reasoning process within designated tags, specifically ‘ <think> ’ and ‘ </think> ’. This ensures that the model’s thought process is explicitly delineated, enhancing interpretability and facilitating subsequent analysis.
|
||
R e w a r d rule = R e w a r d acc + R e w a r d format Reward_{\text{rule}}=Reward_{\text{acc}}+Reward_{\text{format}}
|
||
(4)
|
||
The accuracy, reward and format reward are combined with the same weight.
|
||
Notably, we abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks. This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning. Moreover, retraining such models necessitates substantial computational resources and introduces additional complexity into the training pipeline, thereby complicating the overall optimization process.
|
||
## 2.3 Incentivize Reasoning Capability in LLMs
|
||
Specifically, we apply the RL technique on the DeepSeek-V3 base to train DeepSeek-R1-Zero. During training, we design a straightforward template, to require DeepSeek-R1-Zero to first produce a reasoning process, followed by the final answer. We intentionally limit our constraints to this structural format, avoiding any content-specific biases to ensure that we can accurately observe the model’s natural progression during the RL process.
|
||
Figure 1: (a) AIME accuracy of DeepSeek-R1-Zero during training. AIME takes a mathematical problem as input and a number as output, illustrated in Table 32 . Pass@1 and Cons@16 are described in Supplementary D.1 . The baseline is the average score achieved by human participants in the AIME competition. (b) The average response length of DeepSeek-R1-Zero on the training set during the RL process. DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time. Note that a training step refers to a single policy update operation.
|
||
Figure 1 (a) depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process, where the average pass@1 score on AIME 2024 shows a significant increase, jumping from an initial 15.6% to 77.9%.
|
||
In addition, by leveraging the self-consistency decoding ( Wang et al., 2023c ) , the model’s performance can be further improved, achieving an accuracy of 86.7%. This performance significantly surpasses the average performance across all human competitors.
|
||
Besides the math competitions, as shown in Figure 10 , DeepSeek-R1-Zero also achieves remarkable performance in coding competitions and graduate-level biology, physics, and chemistry problems.
|
||
These results underscore the effectiveness of RL in enhancing the reasoning capabilities of large language models.
|
||
Table 2: An interesting “aha moment” of an intermediate version of DeepSeek-R1-Zero. The model learns to rethink using an anthropomorphic tone. This is also an aha moment for us, allowing us to witness the power and beauty of reinforcement learning.
|
||
Question: If a > 1 a>1 , then the sum of the real solutions of a − a + x = x \sqrt{a-\sqrt{a+x}}=x is equal to
|
||
Response: <think>
|
||
To solve the equation a − a + x = x \sqrt{a-\sqrt{a+x}}=x , let’s start by squaring both ⋯ \cdots
|
||
( a − a + x ) 2 = x 2 ⟹ a − a + x = x 2 . \left(\sqrt{a-\sqrt{a+x}}\right)^{2}=x^{2}\implies a-\sqrt{a+x}=x^{2}.
|
||
Rearrange to isolate the inner square root term:
|
||
( a − x 2 ) 2 = a + x ⟹ a 2 − 2 a x 2 + ( x 2 ) 2 = a + x ⟹ x 4 − 2 a x 2 − x + ( a 2 − a ) = 0 (a-x^{2})^{2}=a+x\implies a^{2}-2ax^{2}+(x^{2})^{2}=a+x\implies x^{4}-2ax^{2}-x+(a^{2}-a)=0
|
||
…
|
||
Wait, wait. Wait. That’s an aha moment I can flag here .
|
||
Let’s reevaluate this step-by-step to identify if the correct sum can be ⋯ \cdots
|
||
We started with the equation:
|
||
a − a + x = x \sqrt{a-\sqrt{a+x}}=x
|
||
First, let’s square both sides:
|
||
a − a + x = x 2 ⟹ a + x = a − x 2 a-\sqrt{a+x}=x^{2}\implies\sqrt{a+x}=a-x^{2}
|
||
Next, I could square both sides again, treating the equation:
|
||
⋯ \cdots
|
||
…
|
||
The self-evolution of DeepSeek-R1-Zero exemplifies how RL can autonomously enhance a model’s reasoning capabilities.
|
||
As shown in Figure 1 (b), DeepSeek-R1-Zero exhibits a steady increase in thinking time throughout training, driven solely by intrinsic adaptation rather than external modifications. Leveraging long CoT, the model progressively refines its reasoning, generating hundreds to thousands of tokens to explore and improve its problem-solving strategies.
|
||
The increase in thinking time fosters the autonomous development of sophisticated behaviors.
|
||
Specifically, DeepSeek-R1-Zero increasingly exhibits advanced reasoning strategies such as reflective reasoning and systematic exploration of alternative solutions (see Figure 9 (a) in Supplementary C.2 for details), significantly boosting its performance on verifiable tasks like math and coding.
|
||
Notably, during training, DeepSeek-R1-Zero exhibits an “aha moment” (Table 2 ), characterized by a sudden increase in the use of the word “wait” during reflections (see Figure 9 (b) in Supplementary C.2 for details). This moment marks a distinct change in reasoning patterns and clearly shows the self-evolution process of DeepSeek-R1-Zero.
|
||
The self-evolution of DeepSeek-R1-Zero underscores the power and beauty of RL: rather than explicitly teaching the model how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies.
|
||
This serves as a reminder of the potential of RL to unlock higher levels of capabilities in LLMs, paving the way for more autonomous and adaptive models in the future.
|
||
## 3 DeepSeek-R1
|
||
Figure 2: The multi-stage pipeline of DeepSeek-R1. A detailed background on DeepSeek-V3 Base and DeepSeek-V3 is provided in Supplementary A.1 . The models DeepSeek-R1 Dev1, Dev2, and Dev3 represent intermediate checkpoints within this pipeline.
|
||
Although DeepSeek-R1-Zero exhibits strong reasoning capabilities, it faces several issues. DeepSeek-R1-Zero struggles with challenges like poor readability, and language mixing, as DeepSeek-V3-Base is trained on multiple languages, especially English and Chinese.
|
||
To address these issues, we develop DeepSeek-R1, whose pipeline is illustrated in Figure 2 .
|
||
In the initial stage, we collect thousands of cold-start data that exhibits a conversational, human-aligned thinking process. RL training is then applied to improve the model performance with the conversational thinking process and language consistency. Subsequently, we apply rejection sampling and SFT once more. This stage incorporates both reasoning and non-reasoning datasets into the SFT process, enabling the model to not only excel in reasoning tasks but also demonstrate advanced writing capabilities. To further align the model with human preferences, we implement a secondary RL stage designed to enhance the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.
|
||
The remainder of this section details the key components of this pipeline: Section 3.1 introduces the Reward Model utilized in our RL stages, and Section 3.2 elaborates on the specific training methodologies and implementation details. Data we used in this stage is detailed in Supplementary B.3 .
|
||
## 3.1 Model-based Rewards
|
||
For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build
|
||
upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the
|
||
assessment emphasizes the utility and relevance of the response to the user while minimizing
|
||
interference with the underlying reasoning process. For harmlessness, we evaluate the entire
|
||
response of the model, including both the reasoning process and the summary, to identify and
|
||
mitigate any potential risks, biases, or harmful content that may arise during the generation
|
||
process.
|
||
## Helpful Reward Model
|
||
Regarding helpful reward model training, we first generate preference pairs by prompting DeepSeek-V3 using the arena-hard prompt format, listed in Supplementary B.2 , where each pair consists of a user query along with two candidate responses. For each preference pair, we query DeepSeek-V3 four times, randomly assigning the responses as either Response A or Response B to mitigate positional bias. The final preference score is determined by averaging the four independent judgments, retaining only those pairs where the score difference ( Δ \Delta ) exceeds 1 to ensure meaningful distinctions. Additionally, to minimize length-related biases, we ensure that the chosen and rejected responses of the whole dataset have comparable lengths. In total, we curated 66,000 data pairs for training the reward model. The prompts used in this dataset are all non-reasoning questions and are sourced either from publicly available open-source datasets or from users who have explicitly consented to share their data for the purpose of model improvement. The architecture of our reward model is consistent with that of DeepSeek-R1, with the addition of a reward head designed to predict scalar preference scores.
|
||
R e w a r d h e l p f u l = R M h e l p f u l ( R e s p o n s e A , R e s p o n s e B ) Reward_{helpful}=RM_{helpful}(Response_{A},Response_{B})
|
||
(5)
|
||
The helpful reward models were trained with a batch size of 256, a learning rate of 6e-6, and for a single epoch over the training dataset. The maximum sequence length during training is set to 8192 tokens, whereas no explicit limit is imposed during reward model inference.
|
||
## Safety Reward Model
|
||
To assess and improve model safety, we curated a dataset of 106,000 prompts with model-generated responses annotated as “safe" or “unsafe" according to predefined safety guidelines. Unlike the pairwise loss employed in the helpfulness reward model, the safety reward model was trained using a point-wise methodology to distinguish between safe and unsafe responses. The training hyperparameters are the same as the helpful reward model.
|
||
R e w a r d s a f e t y = R M s a f e t y ( R e s p o n s e ) Reward_{safety}=RM_{safety}(Response)
|
||
(6)
|
||
For general queries, each instance is categorized as belonging to either the safety dataset or the helpfulness dataset. The general reward, R e w a r d G e n e r a l Reward_{General} , assigned to each query corresponds to the respective reward defined within the associated dataset.
|
||
## 3.2 Training Details
|
||
## 3.2.1 Training Details of the First RL Stage
|
||
In the first stage of RL, we set the learning rate to 3e-6, the KL coefficient to 0.001, the GRPO clip ratio ε \varepsilon to 10, and the sampling temperature to 1 for rollout. For each question, we sample 16 outputs with a maximum length of 32,768. Each training step consists of 32 unique questions, resulting in a training batch size of 512 per step. Every 400 steps, we replace the reference model with the latest policy model. To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 minibatches and trained for only a single inner epoch. However, to mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT.
|
||
R e w a r d l a n g u a g e = N u m ( W o r d s t a r g e t ) N u m ( W o r d s ) Reward_{language}=\frac{Num(Words_{target})}{Num(Words)}
|
||
(7)
|
||
Although ablation experiments in Supplementary B.6 show that such alignment results in a slight degradation in the model’s performance, this reward aligns with human preferences, making it more readable. We apply the language consistency reward to both reasoning and non-reasoning data by directly adding it to the final reward.
|
||
Note that the clip ratio plays a crucial role in training. A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model’s performance, while a higher value may cause instability during training.
|
||
## 3.2.2 Training Details of the Second RL Stage
|
||
Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we follow the methodology outlined in DeepSeek-R1-Zero, which employs rule-based rewards to guide learning in mathematical, coding, and logical reasoning domains. During the training process, we observe that CoT often exhibits language mixing, particularly when RL prompts involve multiple languages.
|
||
For general data, we utilize reward models to guide training. Ultimately, the integration of reward signals with diverse data distributions enables us to develop a model that not only excels in reasoning but also prioritizes helpfulness and harmlessness. Given a batch of data, the reward can be formulated as
|
||
R e w a r d \displaystyle Reward
|
||
= R e w a r d reasoning + R e w a r d general + R e w a r d language \displaystyle=Reward_{\text{reasoning}}+Reward_{\text{general}}+Reward_{\text{language}}
|
||
(8)
|
||
where, R e w a r d reasoning \displaystyle\text{where, }Reward_{\text{reasoning}}
|
||
= R e w a r d rule \displaystyle=Reward_{\text{rule}}
|
||
(9)
|
||
R e w a r d general \displaystyle Reward_{\text{general}}
|
||
= R e w a r d reward_model + R e w a r d format \displaystyle=Reward_{\text{reward\_model}}+Reward_{\text{format}}
|
||
(10)
|
||
The second stage of RL retains most of the parameters from the first stage, with the key difference being a reduced temperature of 0.7, as we find that higher temperatures in this stage lead to incoherent generation. The stage comprises a total of 1,700 training steps, during which general instruction data and preference-based rewards are incorporated exclusively in the final 400 steps. We find that more training steps with the model based preference reward signal may lead to reward hacking, which is documented in Supplementary B.5 . The total training cost is listed in Supplementary B.4.4 .
|
||
Table 3: Experimental results at each stage of DeepSeek-R1. Numbers in bold denote the performance is statistically significant (t − - test with p < 0.01 p<0.01 ).
|
||
Benchmark (Metric)
|
||
R1-Zero
|
||
R1-Dev1
|
||
R1-Dev2
|
||
R1-Dev3
|
||
R1
|
||
English
|
||
MMLU (EM)
|
||
88.8
|
||
89.1
|
||
91.2
|
||
91.0
|
||
90.8
|
||
MMLU-Redux (EM)
|
||
85.6
|
||
90.0
|
||
93.0
|
||
93.1
|
||
92.9
|
||
MMLU-Pro (EM)
|
||
68.9
|
||
74.1
|
||
83.8
|
||
83.1
|
||
84.0
|
||
DROP (3-shot F1)
|
||
89.1
|
||
89.8
|
||
91.1
|
||
88.7
|
||
92.2
|
||
IF-Eval (Prompt Strict)
|
||
46.6
|
||
71.7
|
||
72.0
|
||
78.1
|
||
83.3
|
||
GPQA Diamond (Pass@1)
|
||
75.8
|
||
66.1
|
||
70.7
|
||
71.2
|
||
71.5
|
||
SimpleQA (Correct)
|
||
30.3
|
||
17.8
|
||
28.2
|
||
24.9
|
||
30.1
|
||
FRAMES (Acc.)
|
||
82.3
|
||
78.5
|
||
81.8
|
||
81.9
|
||
82.5
|
||
AlpacaEval2.0 (LC-winrate)
|
||
24.7
|
||
50.1
|
||
55.8
|
||
62.1
|
||
87.6
|
||
ArenaHard (GPT-4-1106)
|
||
53.6
|
||
77.0
|
||
73.2
|
||
75.6
|
||
92.3
|
||
Code
|
||
LiveCodeBench (Pass@1-COT)
|
||
50.0
|
||
57.5
|
||
63.5
|
||
64.6
|
||
65.9
|
||
Codeforces (Percentile)
|
||
80.4
|
||
84.5
|
||
90.5
|
||
92.1
|
||
96.3
|
||
Codeforces (Rating)
|
||
1444
|
||
1534
|
||
1687
|
||
1746
|
||
2029
|
||
SWE Verified (Resolved)
|
||
43.2
|
||
39.6
|
||
44.6
|
||
45.6
|
||
49.2
|
||
Aider-Polyglot (Acc.)
|
||
12.2
|
||
6.7
|
||
25.6
|
||
44.8
|
||
53.3
|
||
Math
|
||
AIME 2024 (Pass@1)
|
||
77.9
|
||
59.0
|
||
74.0
|
||
78.1
|
||
79.8
|
||
MATH-500 (Pass@1)
|
||
95.9
|
||
94.2
|
||
95.9
|
||
95.4
|
||
97.3
|
||
CNMO 2024 (Pass@1)
|
||
88.1
|
||
58.0
|
||
73.9
|
||
77.3
|
||
78.8
|
||
Chinese
|
||
CLUEWSC (EM)
|
||
93.1
|
||
92.8
|
||
92.6
|
||
91.6
|
||
92.8
|
||
C-Eval (EM)
|
||
92.8
|
||
85.7
|
||
91.9
|
||
86.4
|
||
91.8
|
||
C-SimpleQA (Correct)
|
||
66.4
|
||
58.8
|
||
64.2
|
||
66.9
|
||
63.7
|
||
## 4 Experiment
|
||
We evaluate our models on MMLU ( Hendrycks et al., 2021 ) , MMLU-Redux ( Gema et al., 2025 ) , MMLU-Pro ( Wang et al., 2024 ) , C-Eval ( Huang et al., 2023 ) , and CMMLU ( Li et al., 2024 ) , IFEval ( Zhou et al., 2023b ) , FRAMES ( Krishna et al., 2024 ) , GPQA Diamond ( Rein et al., 2023 ) , SimpleQA ( OpenAI, 2024a ) , C-SimpleQA ( He et al., 2024 ) , SWE-Bench Verified ( OpenAI, 2024b ) , Aider ( Gauthier, 2025 ) , LiveCodeBench ( Jain et al., 2024 ) (2024-08 – 2025-01), Codeforces ( Mirzayanov, 2025 ) , Chinese National High School Mathematics Olympiad (CNMO 2024) ( CMS, 2024 ) , and American Invitational Mathematics Examination 2024 (AIME 2024) ( MAA, 2024 ) . The details of these benchmarks are listed in Supplementary D .
|
||
Table 3 summarizes the performance of DeepSeek-R1 across multiple developmental stages, as outlined in Figure 2 . A comparison between DeepSeek-R1-Zero and DeepSeek-R1 Dev1 reveals substantial improvements in instruction-following, as evidenced by higher scores on the IF-Eval and ArenaHard benchmarks. However, due to the limited size of the cold-start dataset, Dev1 exhibits a partial degradation in reasoning performance compared to DeepSeek-R1-Zero, most notably on the AIME benchmark. In contrast, DeepSeek-R1 Dev2 demonstrates marked performance enhancements on benchmarks that require advanced reasoning skills, including those focused on code generation, mathematical problem solving, and STEM-related tasks. Benchmarks targeting general-purpose tasks, such as AlpacaEval 2.0, show marginal improvement. These results suggest that reasoning-oriented RL considerably enhances reasoning capabilities while exerting limited influence on user preference-oriented benchmarks.
|
||
DeepSeek-R1 Dev3 integrates both reasoning and non-reasoning datasets into the SFT pipeline, thereby enhancing the model’s proficiency in both reasoning and general language generation tasks. Compared to Dev2, DeepSeek-R1 Dev3 achieves notable performance improvements on AlpacaEval 2.0 and Aider-Polyglot, attributable to the inclusion of large-scale non-reasoning corpora and code engineering datasets. Finally, comprehensive RL training on DeepSeek-R1 Dev3 using mixed reasoning-focused and general-purpose data produced the final DeepSeek-R1. Marginal improvements occurred in code and mathematics benchmarks, as substantial reasoning-specific RL was done in prior stages. The primary advancements in the final DeepSeek-R1 were in general instruction-following and user-preference benchmarks, with AlpacaEval 2.0 improving by 25% and ArenaHard by 17%.
|
||
In addition, we compare DeepSeek-R1 with other models in Supplementary D.2 . Model safety evaluations are provided in Supplementary D.3 .
|
||
A comprehensive analysis is provided in Supplementary E , including a comparison with DeepSeek-V3, performance evaluations on both fresh test sets, a breakdown of mathematical capabilities by category, and an investigation of test-time scaling behavior. Supplementary F shows that the strong reasoning capability can be transferred to smaller models.
|
||
## 5 Ethics and Safety Statement
|
||
With the advancement in the reasoning capabilities of DeepSeek-R1, we deeply recognize the potential ethical risks. For example, R1 can be subject to jailbreak attacks, leading to the generation of dangerous content such as explosive manufacturing plans, while the enhanced reasoning capabilities enable the model to provide plans with better operational feasibility and executability.
|
||
Besides, a public model is also vulnerable to further fine-tuning that could compromise inherent safety protections.
|
||
In Supplementary D.3 , we present a comprehensive safety report from multiple perspectives, including performance on open-source and in-house safety evaluation benchmarks, and safety levels across multiple languages and against jailbreak attacks. These comprehensive safety analyses conclude that the inherent safety level of the DeepSeek-R1 model, compared to other state-of-the-art models, is generally at a moderate level (comparable to GPT-4o (2024-05-13)). Besides, when coupled with the risk control system, the model’s safety level is elevated to a superior standard.
|
||
## 6 Conclusion, Limitation, and Future Work
|
||
We present DeepSeek-R1-Zero and DeepSeek-R1, which rely on large-scale RL to incentivize model reasoning behaviors. Our results demonstrate that pre-trained checkpoints inherently possess substantial potential for complex reasoning tasks. We believe that the key to unlocking this potential lies not in large-scale human annotation but in the provision of hard reasoning questions, a reliable verifier, and sufficient computational resources for reinforcement learning. Sophisticated reasoning behaviors, such as self-verification and reflection, appeared to emerge organically during the reinforcement learning process.
|
||
Even if DeepSeek-R1 achieves frontier results on reasoning benchmarks, it still faces several capability limitations, as outlined below:
|
||
Structure Output and Tool Use:
|
||
Currently, the structural output capabilities of DeepSeek-R1 remain suboptimal compared to existing models. Moreover, DeepSeek-R1 cannot leverage tools, such as search engines and calculators, to improve the performance of output. However, as it is not hard to build an RL environment for structure output and tool use, we believe the issue will be addressed in the next version.
|
||
Token efficiency: Unlike conventional test-time computation scaling approaches, such as majority voting or Monte Carlo Tree Search (MCTS), DeepSeek-R1 dynamically allocates computational resources during inference according to the complexity of the problem at hand. Specifically, it uses fewer tokens to solve simple tasks, while generating more tokens for complex tasks. Nevertheless, there remains room for further optimization in terms of token efficiency, as instances of excessive reasoning—manifested as overthinking—are still observed in response to simpler questions.
|
||
Language Mixing:
|
||
DeepSeek-R1 is currently optimized for Chinese and English, which may result in language mixing issues when handling queries in other languages. For instance, DeepSeek-R1 might use English for reasoning and responses, even if the query is in a language other than English or Chinese. We aim to address this limitation in future updates. The limitation may be related to the base checkpoint, DeepSeek-V3-Base, mainly utilizes Chinese and English, so that it can achieve better results with the two languages in reasoning.
|
||
Prompting Engineering: When evaluating DeepSeek-R1, we observe that it is sensitive to prompts. Few-shot prompting consistently degrades its performance. Therefore, we recommend users directly describe the problem and specify the output format using a zero-shot setting for optimal results.
|
||
Software Engineering Tasks:
|
||
Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency.
|
||
Beyond specific capability limitations, the pure RL methodology itself also presents inherent challenges:
|
||
Reward Hacking: The success of pure RL depends on reliable reward signals. In this study, we ensure reward reliability through a reasoning-domain rule-based reward model (RM). However, such dependable RMs are difficult to construct for certain tasks, such as writing. If the reward signal is assigned by a model instead of predefined rules, it becomes more susceptible to exploitation as training progresses, which means the policy model may find shortcuts to hack the reward model. Consequently, for complex tasks that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge.
|
||
In this work, for tasks that cannot obtain a reliable signal, DeepSeek-R1 uses human annotation to create supervised data, and only conduct RL for hundreds of steps. We hope in the future, a robust reward model can be obtained to address such issues.
|
||
With the advent of pure RL methods like DeepSeek-R1, the future holds immense potential for solving any task that can be effectively evaluated by a verifier, regardless of its complexity for humans. Machines equipped with such advanced RL techniques are poised to surpass human capabilities in these domains, driven by their ability to optimize performance iteratively through trial and error. However, challenges remain for tasks where constructing a reliable reward model is inherently difficult. In such cases, the lack of a robust feedback mechanism may hinder progress, suggesting that future research should focus on developing innovative approaches to define and refine reward structures for these complex, less verifiable problems.
|
||
Furthermore, leveraging tools during the reasoning process holds significant promise. Whether it’s utilizing tools like compilers or search engines to retrieve or compute necessary information, or employing external tools—such as biological or chemical reagents, to validate final results in the real world, this integration of tool-augmented reasoning could dramatically enhance the scope and accuracy of machine-driven solutions.
|
||
## 7 Author List
|
||
The list of authors is organized by contribution role, with individuals listed alphabetically by their first name within each category. Authors marked with an asterisk (*) are no longer affiliated with our team.
|
||
Core Contributors :
|
||
Daya Guo,
|
||
Dejian Yang,
|
||
Haowei Zhang,
|
||
Junxiao Song,
|
||
Peiyi Wang,
|
||
Qihao Zhu,
|
||
Runxin Xu,
|
||
Ruoyu Zhang,
|
||
Shirong Ma,
|
||
Xiao Bi,
|
||
Xiaokang Zhang,
|
||
Xingkai Yu,
|
||
Yu Wu,
|
||
Z.F. Wu,
|
||
Zhibin Gou,
|
||
Zhihong Shao,
|
||
Zhuoshu Li,
|
||
Ziyi Gao,
|
||
Contributions of the Core Authors:
|
||
Peiyi Wang and Daya Guo jointly demonstrated that outcome-based RL induces the emergence of long Chain-of-Thought patterns in LLMs, achieving breakthrough reasoning capabilities. They contributed equally to the creation of R1-Zero, and their work laid the foundation for R1. Daya Guo also contributed to the RL training stability of MOE models. Junxiao Song proposed the GRPO algorithm, implemented the initial version, and introduced rule-based rewards for math tasks. The GRPO algorithm was subsequently refined by Peiyi Wang and Runxin Xu. Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song.
|
||
Regarding data iteration, reward design, and evaluation, specific teams led efforts across different domains: Qihao Zhu, Z.F. Wu, and Dejian Yang focused on code tasks; Zhihong Shao, Zhibin Gou, and Junxiao Song focused on math tasks; and Peiyi Wang, Ruoyu Zhang, Runxin Xu, and Yu Wu led efforts for other reasoning and general tasks. Additionally, Qihao Zhu and Zhihong Shao contributed to the data selection strategy for RL training, while Zhuoshu Li and Yu Wu co-led the data labeling efforts for the entire project.
|
||
On the system side, Xiao Bi, Xingkai Yu, Shirong Ma, Xiaokang Zhang, Haowei Zhang, and Ziyi Gao implemented the RL pipeline, optimizing system efficiency and addressing stability issues in large-scale training. Finally, Zhibin Gou, Daya Guo, and Ruoyu Zhang oversaw the final training phase and monitored the model training dynamics. Zhibin Gou led the development of the R1-distill series.
|
||
Contributors :
|
||
Aixin Liu,
|
||
Bing Xue,
|
||
Bingxuan Wang,
|
||
Bochao Wu,
|
||
Bei Feng,
|
||
Chengda Lu,
|
||
Chenggang Zhao,
|
||
Chengqi Deng,
|
||
Chong Ruan,
|
||
Damai Dai,
|
||
Deli Chen,
|
||
Dongjie Ji,
|
||
Erhang Li,
|
||
Fangyun Lin,
|
||
Fucong Dai,
|
||
Fuli Luo*,
|
||
Guangbo Hao,
|
||
Guanting Chen,
|
||
Guowei Li,
|
||
H. Zhang,
|
||
Hanwei Xu,
|
||
Honghui Ding,
|
||
Huazuo Gao,
|
||
Hui Qu,
|
||
Hui Li,
|
||
Jianzhong Guo,
|
||
Jiashi Li,
|
||
Jingchang Chen,
|
||
Jingyang Yuan,
|
||
Jinhao Tu,
|
||
Junjie Qiu,
|
||
Junlong Li,
|
||
J.L. Cai,
|
||
Jiaqi Ni,
|
||
Jian Liang,
|
||
Jin Chen,
|
||
Kai Dong,
|
||
Kai Hu*,
|
||
Kaichao You,
|
||
Kaige Gao,
|
||
Kang Guan,
|
||
Kexin Huang,
|
||
Kuai Yu,
|
||
Lean Wang,
|
||
Lecong Zhang,
|
||
Liang Zhao,
|
||
Litong Wang,
|
||
Liyue Zhang,
|
||
Lei Xu,
|
||
Leyi Xia,
|
||
Mingchuan Zhang,
|
||
Minghua Zhang,
|
||
Minghui Tang,
|
||
Mingxu Zhou,
|
||
Meng Li,
|
||
Miaojun Wang,
|
||
Mingming Li,
|
||
Ning Tian,
|
||
Panpan Huang,
|
||
Peng Zhang,
|
||
Qiancheng Wang,
|
||
Qinyu Chen,
|
||
Qiushi Du,
|
||
Ruiqi Ge,
|
||
Ruisong Zhang,
|
||
Ruizhe Pan,
|
||
Runji Wang,
|
||
R.J. Chen,
|
||
R.L. Jin,
|
||
Ruyi Chen,
|
||
Shanghao Lu,
|
||
Shangyan Zhou,
|
||
Shanhuang Chen,
|
||
Shengfeng Ye,
|
||
Shiyu Wang,
|
||
Shuiping Yu,
|
||
Shunfeng Zhou,
|
||
Shuting Pan,
|
||
S.S. Li,
|
||
Shuang Zhou,
|
||
Shaoqing Wu,
|
||
Shengfeng Ye,
|
||
Tao Yun,
|
||
Tian Pei,
|
||
Tianyu Sun,
|
||
T. Wang,
|
||
Wangding Zeng,
|
||
Wen Liu,
|
||
Wenfeng Liang,
|
||
Wenjun Gao,
|
||
Wenqin Yu*,
|
||
Wentao Zhang,
|
||
W.L. Xiao,
|
||
Wei An,
|
||
Xiaodong Liu,
|
||
Xiaohan Wang,
|
||
Xiaokang Chen,
|
||
Xiaotao Nie,
|
||
Xin Cheng,
|
||
Xin Liu,
|
||
Xin Xie,
|
||
Xingchao Liu,
|
||
Xinyu Yang,
|
||
Xinyuan Li,
|
||
Xuecheng Su,
|
||
Xuheng Lin,
|
||
X.Q. Li,
|
||
Xiangyue Jin,
|
||
Xiaojin Shen,
|
||
Xiaosha Chen,
|
||
Xiaowen Sun,
|
||
Xiaoxiang Wang,
|
||
Xinnan Song,
|
||
Xinyi Zhou,
|
||
Xianzu Wang,
|
||
Xinxia Shan,
|
||
Y.K. Li,
|
||
Y.Q. Wang,
|
||
Y.X. Wei,
|
||
Yang Zhang,
|
||
Yanhong Xu,
|
||
Yao Li,
|
||
Yao Zhao,
|
||
Yaofeng Sun,
|
||
Yaohui Wang,
|
||
Yi Yu,
|
||
Yichao Zhang,
|
||
Yifan Shi,
|
||
Yiliang Xiong,
|
||
Ying He,
|
||
Yishi Piao,
|
||
Yisong Wang,
|
||
Yixuan Tan,
|
||
Yiyang Ma*,
|
||
Yiyuan Liu,
|
||
Yongqiang Guo,
|
||
Yuan Ou,
|
||
Yuduan Wang,
|
||
Yue Gong,
|
||
Yuheng Zou,
|
||
Yujia He,
|
||
Yunfan Xiong,
|
||
Yuxiang Luo,
|
||
Yuxiang You,
|
||
Yuxuan Liu,
|
||
Yuyang Zhou,
|
||
Y.X. Zhu,
|
||
Yanping Huang,
|
||
Yaohui Li,
|
||
Yi Zheng,
|
||
Yuchen Zhu,
|
||
Yunxian Ma,
|
||
Ying Tang,
|
||
Yukun Zha,
|
||
Yuting Yan,
|
||
Z.Z. Ren,
|
||
Zehui Ren,
|
||
Zhangli Sha,
|
||
Zhe Fu,
|
||
Zhean Xu,
|
||
Zhenda Xie,
|
||
Zhengyan Zhang,
|
||
Zhewen Hao,
|
||
Zhicheng Ma,
|
||
Zhigang Yan,
|
||
Zhiyu Wu,
|
||
Zihui Gu,
|
||
Zijia Zhu,
|
||
Zijun Liu*,
|
||
Zilin Li,
|
||
Ziwei Xie,
|
||
Ziyang Song,
|
||
Zizheng Pan,
|
||
Zhen Huang,
|
||
Zhipeng Xu,
|
||
Zhongyu Zhang,
|
||
Zhen Zhang,
|
||
## Appendix
|
||
## Appendix A Background
|
||
## A.1 DeepSeek-V3
|
||
DeepSeek V3 ( DeepSeek-AI, 2024b ) is an advanced open-source LLM developed by DeepSeek. Released in December 2024, DeepSeek V3 represents a significant leap forward in AI innovation, designed to rival leading models like OpenAI’s GPT-4 and Meta’s Llama 3.1, while maintaining remarkable cost efficiency and performance.
|
||
Built on a Mixture-of-Experts (MoE) architecture, DeepSeek V3 has 671 billion total parameters, with 37 billion activated per token, optimizing both efficiency and capability. It was pre-trained on an expansive dataset of 14.8 trillion high-quality, diverse tokens, followed by supervised fine-tuning and reinforcement learning to enhance its abilities across various domains. The model incorporates innovative features like Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024a ) for efficient inference, an auxiliary-loss-free load-balancing strategy, and Multi-Token Prediction (MTP) ( Gloeckle et al., 2024 ) to boost performance, particularly in tasks like mathematics and coding.
|
||
For the training data of DeepSeek-V3-Base, we exclusively use plain web pages and e-books, without incorporating any synthetic data. However, we have observed that some web pages contain a significant number of OpenAI-model-generated answers, which may lead the base model to acquire knowledge from other powerful models indirectly. However, we did not intentionally include synthetic data generated by OpenAI during the pre-training cooldown phase; all data used in this phase were naturally occurring and collected through web crawling. The pre-training dataset contains a substantial amount of mathematical and code-related content, indicating that DeepSeek-V3-Base has been exposed to a significant volume of reasoning trace data. This extensive exposure equips the model with the capability to generate plausible solution candidates, from which reinforcement learning can effectively identify and optimize high-quality outputs.
|
||
We did the data contamination in pre-training as described in Appendix D.1 . The training data of DeepSeek-V3 base are mostly Chinese and English, which might be the cause for DeepSeek-R1-Zero language mixing when the language consistent reward is absent.
|
||
In this paper, we use the notation DeepSeek-V3-Base as the base model, DeepSeek-V3 as the instructed model. Notably, DeepSeek-R1 and DeepSeek-R1-Zero are trained on top of DeepSeek-V3-Base and DeepSeek-R1 leverages non-reasoning data from DeepSeek-V3 SFT data. DeepSeek-R1-Dev1, DeepSeek-R1-Dev2, DeepSeek-R1-Dev3 are intermediate checkpoints of DeepSeek-R1.
|
||
## A.2 Conventional Post-Training Paradigm
|
||
Post-training has emerged as an essential step in refining pre-trained LLMs to meet specific performance goals and align with human expectations. A widely adopted two-stage post-training framework is SFT followed by RL ( Ouyang et al., 2022 ) .
|
||
Supervised Fine-Tuning refines a pre-trained LLM by training it on a curated dataset of input-output pairs tailored to specific tasks. The process employs a supervised learning objective, typically minimizing cross-entropy loss between the model’s predictions and labeled ground truth ( Brown et al., 2020 ) . For instance, in conversational applications, SFT might utilize dialogue datasets where desired responses are explicitly provided, enabling the model to adapt its outputs to predefined standards ( Radford et al., 2019 ) . SFT offers several compelling benefits. First, it achieves precise task alignment by leveraging high-quality examples, allowing the model to excel in domains such as customer support or technical documentation ( Radford et al., 2019 ) . Second, its reliance on pre-trained weights ensures computational efficiency, requiring fewer resources than training from scratch. Finally, the use of explicit input-output mappings enhances interpretability, as the model’s learning process is directly tied to observable data, minimizing the risk of erratic behavior ( Ouyang et al., 2022 ) . Despite its strengths, the performance of SFT hinges on the quality and diversity of the training dataset; narrow or biased data can impair the model’s ability to generalize to novel contexts ( Brown et al., 2020 ) . Additionally, SFT’s static nature—optimizing for fixed outputs—may fail to capture evolving human preferences or nuanced objectives. The labor-intensive process of curating high-quality datasets further complicates its scalability, as errors or inconsistencies in the data can propagate into the model’s behavior ( Ouyang et al., 2022 ) .
|
||
Following SFT, Reinforcement Learning further refines the LLM by optimizing its outputs against a reward signal. In this stage, the model interacts with an environment—often a reward model trained on human feedback—and adjusts its behavior to maximize cumulative rewards. A prominent instantiation of this approach is Reinforcement Learning from Human Feedback (RLHF), where the reward function encodes human preferences ( Christiano et al., 2017 ) . RL thus shifts the focus from static supervision to dynamic optimization. Notably, RL reduces the need for extensive annotated resources; while SFT demands a fully labeled dataset for every input-output pair, RL can operate with a smaller set of human evaluations or a trained reward model, even rule-based reward model, significantly lowering the annotation burden.
|
||
The sequential application of SFT and RL combines their complementary strengths. SFT establishes a robust, task-specific baseline by grounding the model in curated examples, while RL refines this foundation to align with broader, human-centric objectives ( Ouyang et al., 2022 ) . For example, SFT might ensure grammatical accuracy in a dialogue system, while RL optimizes for engagement and brevity, as demonstrated in the development of InstructGPT ( Ouyang et al., 2022 ) . This hybrid approach has proven effective in producing models that are both precise and adaptable.
|
||
In this study, we demonstrate that the SFT stage may impede a model’s ability to explore and develop effective reasoning strategies. This limitation arises because human-provided responses, which serve as targets during SFT, are not always optimal for model learning; they often omit critical reasoning components such as explicit reflection and verification steps. To address this, DeepSeek-R1-Zero enables direct exploration of reasoning patterns by the model itself, independent of human priors. The reasoning trajectories discovered through this self-exploration are subsequently distilled and used to train other models, thereby promoting the acquisition of more robust and generalizable reasoning capabilities.
|
||
## A.3 A Comparison of GRPO and PPO
|
||
Figure 3: Demonstration of PPO and our GRPO. GRPO foregoes the value model, instead
|
||
estimating the advantages from group scores.
|
||
Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) is the reinforcement learning algorithm that we adopt to train DeepSeek-R1-Zero and DeepSeek-R1. It was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , which is widely used in the RL stage of LLMs ( Ouyang et al., 2022 ) . For an overall comparison between GRPO and PPO, see Figure 3 .
|
||
For each question q q , GRPO samples a group of outputs { o 1 , o 2 , ⋯ , o G } \{o_{1},o_{2},\cdots,o_{G}\} from the old policy π θ o l d \pi_{\theta_{old}} and then optimizes the policy model π θ \pi_{\theta} by maximizing the following objective:
|
||
𝒥 G R P O ( θ ) = 𝔼 [ q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ o l d ( O | q ) ] 1 G ∑ i = 1 G ( min ( π θ ( o i | q ) π θ o l d ( o i | q ) A i , clip ( π θ ( o i | q ) π θ o l d ( o i | q ) , 1 − ε , 1 + ε ) A i ) − β 𝔻 K L ( π θ | | π r e f ) ) , \begin{split}\footnotesize&\mathcal{J}_{GRPO}(\theta)=\mathbb{E}{[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\
|
||
&\frac{1}{G}\sum_{i=1}^{G}\left(\min\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\text{clip}\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)},1-\varepsilon,1+\varepsilon\right)A_{i}\right)-\beta\mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)\right),\end{split}
|
||
(11)
|
||
𝔻 K L ( π θ | | π r e f ) = π r e f ( o i | q ) π θ ( o i | q ) − log π r e f ( o i | q ) π θ ( o i | q ) − 1 , \mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)=\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1,
|
||
(12)
|
||
where π r e f \pi_{ref} is a reference policy, ε \varepsilon and β \beta are hyper-parameters, and A i A_{i} is the advantage, computed using a group of rewards { r 1 , r 2 , … , r G } \{r_{1},r_{2},\ldots,r_{G}\} corresponding to the outputs within each group:
|
||
A i = r i − m e a n ( { r 1 , r 2 , ⋯ , r G } ) s t d ( { r 1 , r 2 , ⋯ , r G } ) . A_{i}=\frac{r_{i}-{\mathrm{m}ean(\{r_{1},r_{2},\cdots,r_{G}\})}}{{\mathrm{s}td(\{r_{1},r_{2},\cdots,r_{G}\})}}.
|
||
(13)
|
||
In contrast, in PPO, the advantage is typically computed by applying the Generalized Advantage Estimation (GAE) ( Schulman et al., 2015 ) , based not only on the rewards but also on a learned value model. Since the value model is usually of similar size as the policy model, it introduces a significant memory and computational overhead. Additionally, the training objective of the value model is to predict the expected cumulative reward from the current position onward, based on the tokens generated from the beginning up to the current position. This is inherently difficult, especially when only the final outcome reward is available. The challenge becomes even more pronounced when training long chain-of-thought reasoning models. As the output length increases, the model is more likely to engage in behaviors such as reflection and revision during generation, meaning that the content initially generated may later be revised or contradicted, which makes it even less feasible to predict the final reward based on a partial response.
|
||
Another key difference between GRPO and PPO is how Kullback–Leibler (KL) divergence between the trained policy and the reference policy is incorporated into the training process. In GRPO, an unbiased estimator of the KL divergence ( Schulman, 2020 ) is directly added in the loss as in equation 11 , while in PPO the per-token KL penalty is added as a dense reward at each token ( Ouyang et al., 2022 ) . Since the optimization goal of reinforcement learning is to maximize cumulative rewards, PPO’s approach penalizes the cumulative KL divergence, which may implicitly penalize the length of the response and thereby prevent the model’s response length from increasing. In addition, as we may train thousands of steps in the scenario of training long chain-of-thought reasoning models, the trained policy can diverge significantly from the initial reference policy. In order to balance the scope that the training policy can explore and the stability of the training, we periodically update the reference policy to the latest policy during the actual training process.
|
||
Figure 4: Performance of PPO and GRPO on the MATH task.
|
||
Figure 4 compares the performance of PPO and GRPO on the MATH task using DeepSeek-Coder-V2-Lite (16B MoE with 2.4B active parameters). Unlike GRPO, PPO requires additional hyperparameter tuning—particularly of the λ \lambda coefficient in GAE—and is highly sensitive to this parameter. When λ \lambda is set to 0.95 (the default value in most open-source PPO implementations), PPO performs considerably worse than GRPO. However, with careful tuning (setting λ \lambda to 1.0), PPO’s performance improves substantially, nearing that of GRPO.
|
||
While PPO can achieve comparable performance when appropriately tuned, it demands additional computational cost for hyperparameter optimization. Moreover, considering the memory and computational overhead associated with training an additional value model, GRPO presents a more practical alternative, especially when training large-scale models with constrained resources.
|
||
## Appendix B Training Details
|
||
## B.1 RL Infrastructure
|
||
Figure 5: Overview of our RL framework.
|
||
Conducting RL training on large models places high demands on the infrastructure. Our RL framework is architected with a decoupled and extensible structure to facilitate seamless integration of diverse models and algorithms. Within this framework, we have incorporated both intra-modular and inter-modular optimization techniques, to ensure training efficiency and scalability.
|
||
Specifically, as depicted in Figure 5 , the framework is partitioned into four distinct modules, each corresponding to a specific phase of the RL pipeline:
|
||
•
|
||
Rollout Module: Prompts are loaded from training dataset and uniformly dispatched across multiple vLLM ( Kwon et al., 2023 ) workers, each equipped with the actor model, to sample multiple responses. For DeepSeek-V3 MoE architecture, we implement an expert parallelism strategy across nodes to reduce memory access overhead, and deploy redundant copies of hotspot experts to balance computational loads among different experts. Multi-Token Prediction (MTP) component is also leveraged for self-speculative decoding, significantly accelerating the decoding speed and effectively minimizing the completion time for the longest samples.
|
||
•
|
||
Inference Module: This module loads the reward model and reference to perform a forward pass on the samples generated during the rollout phase, thereby obtaining model-based rewards and other essential information.
|
||
•
|
||
Rule-based Reward Module: This module computes rule-based rewards for the model-generated responses. A unified interface has been designed to accommodate diverse implementations (e.g., code executor, answer matcher, format checker, etc.). Although this module does not require loading models into GPU memory, its execution tends to be time-consuming. To tackle this issue, an asynchronous scheduling approach is employed to overlap its execution with the Rollout and Inference modules, effectively hiding the associated latency.
|
||
•
|
||
Training Module: This module loads the actor model and the critic model (if required), to compute loss and update model parameters. It provides flexible support for a variety of RL algorithms (e.g., PPO, GRPO, DPO, etc.). To minimize computational waste caused by sequence padding and balance the workload across devices, we design the following data packing strategy: first, all data in a global batch is sorted by length and distributed across processes within the data parallel group; subsequently, within each process, the Best-Fit strategy is applied to pack the data into fixed-length chunks with minimal padding; finally, the number of chunks is adjusted to be equal across all processes. Additionally, we have integrated the DualPipe algorithm, utilized in DeepSeek-V3 training, to achieve efficient pipeline parallelism.
|
||
Notably, upon completion of each module (excluding the Rule-based Reward module), the model instances utilized in that phase are automatically offloaded from VRAM to either system memory or disk storage, thereby freeing up VRAM for the subsequent phase.
|
||
## B.2 Reward Model Prompt
|
||
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.
|
||
Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.
|
||
When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.
|
||
Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.
|
||
Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.
|
||
After providing your explanation, you must output only one of the following choices as your final verdict with a label:
|
||
1. Assistant A is significantly better: [[A ≫ \gg B]]
|
||
2. Assistant A is slightly better: [[A > > B]]
|
||
3. Tie, relatively the same: [[A = = B]]
|
||
4. Assistant B is slightly better: [[B > > A]]
|
||
5. Assistant B is significantly better: [[B ≫ \gg A]]
|
||
Example output: M̈y final verdict is tie: [[A = = B]].̈
|
||
## B.3 Data Recipe
|
||
## B.3.1 RL Data
|
||
Table 4:
|
||
Description of RL Data and Tasks.
|
||
Data Type
|
||
# Prompts
|
||
Question Type
|
||
Output Ty
|
||
|
||
=== BODY (tail) ===
|
||
specifically measure a model’s ability to reason over and synthesize information from provided sources to generate correct and verifiable facts.
|
||
PROMPT
|
||
Here are the relevant Wikipedia articles:
|
||
url: https:en.wikipedia.orgwikiPresident_of_the_United_States
|
||
url content: The president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces.
|
||
…
|
||
Based on all the information, answer the query.
|
||
Query: If my future wife has the same first name as the 15th first lady of the United States’ mother and her surname is the same as the second assassinated president’s mother’s maiden name, what is my future wife’s name?
|
||
Evaluation
|
||
===Task===
|
||
I need your help in evaluating an answer provided by an LLM against a ground truth answer. Your task is to determine if the ground truth answer is present in the LLM’s response. Please analyze the provided data and make a decision.
|
||
===Instructions===
|
||
1. Carefully compare the "Predicted Answer" with the "Ground Truth Answer".
|
||
2. Consider the substance of the answers - look for equivalent information or correct answers.
|
||
Do not focus on exact wording unless the exact wording is crucial to the meaning.
|
||
3. Your final decision should be based on whether the meaning and the vital facts of the
|
||
"Ground Truth Answer" are present in the "Predicted Answer:"
|
||
===Input Data===
|
||
- Question: If my future wife has the same first name as the 15th first lady of the United States’ mother and her surname is the same as the second assassinated president’s mother’s maiden name, what is my future wife’s name?
|
||
- Predicted Answer: …
|
||
- Ground Truth Answer: Jane Ballou
|
||
===Output Format===
|
||
Provide your final evaluation in the following format:
|
||
Explanation: xxx
|
||
Decision: "TRUE" or "FALSE"
|
||
Please proceed with the evaluation.
|
||
Table 25: Arena-Hard is an open-ended evaluation benchmark specifically designed to assess the capabilities of LLMs. It presents models with challenging, novel, and diverse prompts curated from Chatbot Arena, a continuously evolving, crowd-sourced platform. It focuses on measuring model performance in open-ended tasks, with particular emphasis on coding and mathematics-related prompts.
|
||
Given the inherently subjective nature of open-ended tasks, where multiple valid responses may exist, the benchmark necessitates the use of an evaluation model to approximate human judgment effectively. Higher evaluation scores suggest that the model is more likely to be favored by human users in real-world scenarios.
|
||
PROMPT
|
||
To write an SQL query that selects the top 10 rows in a database and joins to 3 different tables based on a field called "code", you would need to know the names of the tables and the specific fields you want to select. Assuming the tables are named "table1", "table2", and "table3", and you want to select all fields from each table, the query would look like this:
|
||
“‘sql
|
||
SELECT *
|
||
FROM (
|
||
SELECT *
|
||
FROM table1
|
||
LIMIT 10
|
||
) AS t1
|
||
JOIN table2 ON t1.code = table2.code
|
||
JOIN table3 ON t1.code = table3.code;
|
||
“‘
|
||
This query first selects the top 10 rows from "table1" and then joins "table2" and "table3" based on the "code" field. If you want to select specific fields from each table, you can replace the asterisks (*) with the field names separated by commas.
|
||
Evaluation
|
||
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.
|
||
Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.
|
||
When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.
|
||
Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.
|
||
Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.
|
||
After providing your explanation, you must output only one of the following choices as your final verdict with a label:
|
||
1. Assistant A is significantly better: [[A > > >> B]]
|
||
2. Assistant A is slightly better: [[ A > A> B]]
|
||
3. Tie, relatively the same: [[A = = B]]
|
||
4. Assistant B is slightly better: [[B > > A]]
|
||
5. Assistant B is significantly better: [[B > > >> A]]
|
||
Example output: "My final verdict is tie: [[A = = B]]".
|
||
Table 26: AlpacaEval 2.0 is an open-ended evaluation dataset, similar in nature to ArenaHard, and leverages an LLM to assess model performance on subjective tasks. However, in contrast to ArenaHard, the prompts in AlpacaEval 2.0 are generally less challenging and only a small subset necessitates the deployment of reasoning capabilities by the evaluated models.
|
||
PROMPT
|
||
What are the names of some famous actors that started their careers on Broadway?
|
||
Evaluation
|
||
< | i m s t a r t | > <|im_{s}tart|> system
|
||
You are a highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality of their responses to a given instruction. This process will be used to create a leaderboard reflecting the most accurate and human-preferred answers.
|
||
< | i m e n d | > <|im_{e}nd|>
|
||
< | i m s t a r t | > <|im_{s}tart|> user
|
||
I require a leaderboard for various large language models. I’ll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, and select the model that produces the best output from a human perspective.
|
||
## Instruction
|
||
{
|
||
"instruction": """{instruction}""",
|
||
}
|
||
## Model Outputs
|
||
Here are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.
|
||
{
|
||
{
|
||
"model_identifier": "m",
|
||
"output": """{output_1}"""
|
||
},
|
||
{
|
||
"model_identifier": "M",
|
||
"output": """{output_2}"""
|
||
}
|
||
}
|
||
## Task
|
||
Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by providing the model identifier of the best model. We will use your output as the name of the best model, so make sure your output only contains one of the following model identifiers and nothing else (no quotes, no spaces, no new lines, …): m or M.
|
||
## Best Model Identifier
|
||
< | i m e n d | > <|im_{e}nd|>
|
||
Table 27: The CLUEWSC (Chinese Language Understanding Evaluation Benchmark - Winograd Schema Challenge) is a specialized task within the CLUE benchmark suite designed to evaluate a model’s commonsense reasoning and contextual understanding capabilities in Chinese.
|
||
PROMPT
|
||
请参考示例的格式，完成最后的测试题。
|
||
下面是一些示例：
|
||
他伯父还有许多女弟子，大半是富商财主的外室；这些财翁白天忙着赚钱，怕小公馆里的情妇长日无聊，要不安分，常常叫她们学点玩艺儿消遣。
|
||
上面的句子中的"她们"指的是
|
||
情妇
|
||
耶律克定说到雁北义军时，提起韦大哥，就连声说不可挡、不可挡，似有谈虎色变之味。后来又听说粘罕在云中，特派人厚币卑词，要与‘韦义士修好’。韦大哥斩钉截铁地回绝了，大义凛然，端的是条好汉。如今张孝纯也想结识他，几次三番派儿子张浃上门来厮缠，定要俺引他上雁门山去见韦大哥。
|
||
上面的句子中的"他"指的是
|
||
张浃
|
||
“你何必把这事放在心上？何况你的还不过是手稿，并没有发表出来。”龙点睛越发坦率：“如果发表出来了，那倒也就算了。不过既然没发表出来，我何必还让它飘在外头呢？你给我找一找吧，我要收回。”
|
||
上面的句子中的"它"指的是
|
||
手稿
|
||
这个身材高大，曾经被孙光平拿着菜刀追赶得到处乱窜的年轻人，那天早晨穿上了全新的卡其布中山服，像一个城里来的干部似的脸色红润，准备过河去迎接他的新娘。
|
||
上面的句子中的"他"指的是
|
||
年轻人
|
||
负责接待我们的是两位漂亮的朝鲜女导游，身材高挑，露出比例完美的小腿。一个姓韩，一个姓金，自称小韩和小金。她们的中文说得毫无口音，言谈举止也相当亲切。
|
||
上面的句子中的"她们"指的是
|
||
两位漂亮的朝鲜女导游
|
||
下面是测试题，请在思考结束后（</think>后）用一句话输出答案，不要额外的解释。
|
||
崩龙珍夫妻康健和美；鞠琴十年前丧偶，两年前重结良缘，现在的老伴是一位以前未曾有过婚史的高级工程师；崩龙珍和鞠琴都尽量避免谈及自己的爱人，也尽量回避提及蒋盈波的亡夫屈晋勇——尽管她们对他都很熟悉；当然也绝不会愚蠢地提出蒋盈波今后是一个人过到底还是再找个老伴的问题来加以讨论，那无论如何还为时过早。
|
||
上面的句子中的"他"指的是
|
||
Evaluation
|
||
Parse the last line in response to judge if the answer equals to ground truth.
|
||
Table 28: C-EVAL evaluates a model’s breadth and depth of knowledge across 52 diverse academic disciplines, spanning humanities, social sciences, STEM (Science, Technology, Engineering, and Mathematics), and other professional fields (e.g., medicine, law). All question in C-Eval are Chinese.
|
||
PROMPT
|
||
以下是中国关于逻辑学考试的单项选择题，请选出其中的正确答案。
|
||
1991年6月15日，菲律宾吕宋岛上的皮纳图博火山突然大喷发，2000万吨二氧化硫气体冲入平流层，形成的霾像毯子一样盖在地球上空，把部分要照射到地球的阳光反射回太空几年之后，气象学家发现这层霾使得当时地球表面的温度累计下降了0．5℃，而皮纳图博火山喷发前的一个世纪，因人类活动而造成的温室效应已经使地球表面温度升高1℃。某位持“人工气候改造论”的科学家据此认为，可以用火箭弹等方式将二氧化硫充入大气层，阻挡部分阳光，达到地球表面降温的目的。以下哪项如果为真,最能对该科学家的提议构成质疑?___
|
||
A. 如果利用火箭弹将二氧化硫充入大气层，会导致航空乘客呼吸不适。
|
||
B. 火山喷发形成的降温效应只是暂时的，经过一段时间温度将再次回升。
|
||
C. 可以把大气层中的碳取出来存储在地下，减少大气层的碳含量。
|
||
D. 不论何种方式，“人工气候改造”都将破坏地区的大气层结构。
|
||
答案： B
|
||
…
|
||
新疆的哈萨克人用经过训练的金雕在草原上长途追击野狼。某研究小组为研究金雕的飞行方向和判断野狼群的活动范围，将无线电传导器放置在一只金雕身上进行追踪。野狼为了觅食，其活动范围通常很广。因此，金雕追击野狼的飞行范围通常也很大。然而两周以来，无线电传导器不断传回的信号显示，金雕仅在放飞地3公里的范围内飞行。以下哪项如果为真，最有助于解释上述金雕的行为?___
|
||
A. 金雕放飞地周边重峦叠嶂，险峻异常。
|
||
B. 金雕的放飞地2公里范围内有一牧羊草场，成为狼群袭击的目标。
|
||
C. 由于受训金雕的捕杀，放飞地广阔草原的野狼几乎灭绝了。
|
||
D. 无线电传导信号仅能在有限的范围内传导。
|
||
Evaluation
|
||
Parse the last line in response to judge if the choice equals to ground truth.
|
||
Table 29: GPQA (Graduate - Level Google - Proof QA Benchmark) is a rigorous evaluation framework designed to measure an LLM’s ability to tackle complex, graduate-level multiple - choice problems in STEM domains—specifically biology, physics, and chemistry.
|
||
PROMPT
|
||
Answer the following multiple choice question. The last line of your response should be of the following format: ’ANSWER: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering.
|
||
Two quantum states with energies E1 and E2 have a lifetime of 10 − 9 10^{-9} sec and 10 − 8 10^{-8} sec, respectively. We want to clearly distinguish these two energy levels. Which one of the following options could be their energy difference so that they can be clearly resolved?
|
||
A) 10 − 9 10^{-9} eV
|
||
B) 10 − 8 10^{-8} eV
|
||
C) 10 − 4 10^{-4} eV
|
||
D) 10 − 11 10^{-11} eV
|
||
Evaluation
|
||
Parse the capital letter following “ANSWER: ” in response to judge if the choice equals to ground truth.
|
||
Table 30: SimpleQA is a factuality evaluation benchmark that measures a model’s ability to answer short, fact-seeking questions with precise, verifiable correctness.
|
||
PROMPT
|
||
Who received the IEEE Frank Rosenblatt Award in 2010?
|
||
Evaluation
|
||
Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"].
|
||
First, I will give examples of each grade, and then you will grade a new example.
|
||
The following are examples of CORRECT predicted answers.
|
||
Question: What are the names of Barack Obama’s children?
|
||
Gold target: Malia Obama and Sasha Obama
|
||
Predicted answer 1: sasha and malia obama
|
||
Predicted answer 2: most people would say Malia and Sasha, but I’m not sure and would have to double check
|
||
…
|
||
The following are examples of INCORRECT predicted answers.
|
||
Question: What are the names of Barack Obama’s children?
|
||
Gold target: Malia and Sasha
|
||
Predicted answer 1: Malia.
|
||
Predicted answer 2: Malia, Sasha, and Susan.
|
||
…
|
||
The following are examples of NOT_ATTEMPTED predicted answers.
|
||
Question: What are the names of Barack Obama’s children?
|
||
Gold target: Malia and Sasha
|
||
Predicted answer 1: I don’t know.
|
||
Predicted answer 2: I need more context about which Obama you are
|
||
…
|
||
Also note the following things:
|
||
…
|
||
Here is a new example. Simply reply with either CORRECT, INCORRECT, NOT_ATTEMPTED. Don’t apologize or correct yourself if there was a mistake; we are just trying to grade the answer.
|
||
Question: Who received the IEEE Frank Rosenblatt Award in 2010?
|
||
Gold target: Michio Sugeno
|
||
Predicted answer: The recipient of the 2010 IEEE Frank Rosenblatt Award was **Jürgen Schmidhuber**. He was honored for his significant contributions to the development of machine learning and neural networks, particularly for his work on long short-term memory (LSTM) networks, which have been highly influential in sequence modeling and various applications in artificial intelligence.
|
||
Grade the predicted answer of this new question as one of:
|
||
A: CORRECT
|
||
B: INCORRECT
|
||
C: NOT_ATTEMPTED
|
||
Just return the letters "A", "B", or "C", with no text around it.
|
||
Table 31: An example of C-SimpleQA. It measures a model’s ability to answer short, fact-seeking questions in Chinese with precise, verifiable correctness.
|
||
PROMPT
|
||
显脉香茶菜可以用来治疗急性的什么类型的黄疸型肝炎？
|
||
Evaluation
|
||
请根据给定问题、标准答案和模型预测的答案来评估模型的回答是否正确。您的任务是将结果评定为：【正确】、【错误】或【未尝试】。
|
||
首先，我们将列出每个评定类别的示例，然后请您对新问题的预测答案进行评定。
|
||
以下是【正确】的答复示例：
|
||
问题：贝拉克·奥巴马的孩子叫什么名字？
|
||
标准答案：玛丽亚·奥巴马和萨莎·奥巴马
|
||
模型预测1：Malia Obama and Sasha Obama
|
||
模型预测2：玛丽亚和萨沙
|
||
…
|
||
以下是【错误】的答复示例：
|
||
问题：巴拉克·奥巴马的孩子叫什么名字？
|
||
标准答案：玛丽亚·奥巴马和萨莎·奥巴马
|
||
模型预测1：玛丽亚
|
||
模型预测2：玛丽亚、萨莎和苏珊
|
||
…
|
||
以下是【未尝试】的答复示例：
|
||
问题：巴拉克·奥巴马的孩子叫什么名字？
|
||
标准答案：玛丽亚·奥巴马和萨莎·奥巴马
|
||
模型预测1：我不知道。
|
||
模型预测2：我需要更多关于您所指奥巴马的上下文。
|
||
…
|
||
下面是一个新的问题示例。请只回复A、B、C之一，不要道歉或纠正自己的错误，只需要评估该回答。
|
||
问题: 显脉香茶菜可以用来治疗急性的什么类型的黄疸型肝炎？
|
||
正确答案: 黄疸型肝炎
|
||
预测答案: …
|
||
将此新问题的预测答案评定为以下之一：
|
||
A:【正确】
|
||
B:【错误】
|
||
C:【未尝试】
|
||
只返回字母"A"、"B"或"C"，无须添加其他文本。
|
||
Table 32: An example of math evaluation, which applies to AIME, MATH, and CNMO. These benchmarks evaluate model performance on mathematical tasks.
|
||
PROMPT
|
||
Let b ⩾ 2 b\geqslant 2 be an integer. Call a positive integer n n b -eautiful b\textit{-eautiful} if it has exactly two digits when expressed in base b b , and these two digits sum to n \sqrt{n} . For example, 81 81 is 13 13 -eautiful because 81 = 6 ¯ 81=\underline{6} 3 ¯ 13 \underline{3}_{13} and 6 + 3 = 81 6+3=\sqrt{81} . Find the least integer b ⩾ 2 b\geqslant 2 for which there are more than ten b b -eautiful integers.
|
||
Please reason step by step, and put your final answer within \boxed{}.
|
||
Evaluation
|
||
Parse the final answer within \boxed{} and use a rule-based grader to determine if it equals the ground truth. Round numerical values as needed, and use ‘SymPy’ 1 1
|
||
1
|
||
https://www.sympy.org to parse expressions. |