TITLE: Inference-Time Scaling for Generalist Reward Modeling
ARXIV: 2504.02495
AUTHORS: Zijun Liu , Peiyi Wang , Runxin Xu , Shirong Ma , Chong Ruan , Peng Li , Yang Liu , Yu Wu
DATE: [Submitted on 3 Apr 2025 ( v1 ), last revised 25 Sep 2025 (this version, v3)]

=== SECTION HEADINGS ===
## Report GitHub Issue
## Inference-Time Scaling for Generalist Reward Modeling
## Abstract
## 1 Introduction
## 2 Preliminaries
## 2.1 Comparisons of Different RM approaches
## Reward Generation Paradigms
## Scoring Patterns
## Representative Methods
## 2.2 Boosting Reward Quality with Principles
## 3 Self-Principled Critique Tuning (SPCT)
## 3.1 Unpinning Principles from Understanding to Generation
## 3.2 Rule-Based Reinforcement Learning
## 4 Inference-Time Scaling with SPCT
## 5 Results on Reward Modeling Benchmarks
## 5.1 Experiment Settings
## 5.2 Results and Analysis
## 6 Related Work
## 7 Conclusion and Future Work
## Ethics Statement

=== BODY (head) ===
Inference-Time Scaling for Generalist Reward Modeling
## Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
2 Preliminaries
2.1 Comparisons of Different RM approaches
Reward Generation Paradigms
Scoring Patterns
Representative Methods
2.2 Boosting Reward Quality with Principles
3 Self-Principled Critique Tuning (SPCT)
3.1 Unpinning Principles from Understanding to Generation
3.2 Rule-Based Reinforcement Learning
4 Inference-Time Scaling with SPCT
5 Results on Reward Modeling Benchmarks
5.1 Experiment Settings
5.2 Results and Analysis
6 Related Work
7 Conclusion and Future Work
References
A Additional Related Work
Constitutional AI
Scalar Reward Models
Semi-Scalar Reward Models
B Limitations and Future Directions
Limitations
Future Directions
C Implementation Details
C.1 Model Training
C.2 Baseline Implementation
D Experiment Details
D.1 Hyper-Parameters
D.2 Benchmarks
D.3 Detailed Results
E Additional Experiments
E.1 Input Flexibility of the Pointwise GRM Approach
E.1.1 Generating Rewards for Many Responses
E.1.2 Generating Rewards for Single Responses
E.1.3 Generating Rewards with Reference
E.2 Transferability of Generated Principles
E.3 Generalization beyond Training Data
E.4 Response Length Analysis for Rule-Based RL
F Qualitative Analysis
F.1 Case Study
F.2 Failure Mode Analysis
G Prompt Templates
License: CC BY-SA 4.0
arXiv:2504.02495v3 [cs.CL] 25 Sep 2025
## Inference-Time Scaling for Generalist Reward Modeling
Zijun Liu
Affiliation: DeepSeek-AI
Email: zj-liu24@mails.tsinghua.edu.cn
Peiyi Wang
Email: wangpeiyi9979@gmail.com
Runxin Xu
Affiliation: DeepSeek-AI
Shirong Ma
Affiliation: DeepSeek-AI
Chong Ruan
Affiliation: DeepSeek-AI
Peng Li
Affiliation: Institute for AI Industry Research (AIR), Tsinghua University
Yang Liu
Affiliation: Dept. of Computer Sci. & Tech., Tsinghua University
Affiliation: Institute for AI Industry Research (AIR), Tsinghua University
Yu Wu
Affiliation: DeepSeek-AI
## Abstract
Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale.
Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability .
A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules.
In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM .
For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and the potential for inference-time scaling.
For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models.
Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide the voting process for better scaling performance.
Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and it can achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope .
1 1 footnotetext: Equal contribution.
† Work done during internship at DeepSeek-AI.
Figure 1: Inference-time scaling performance with different RMs on all tested RM benchmarks. Results are shown with up to 8 samples for each method, and are further scaled to 32 samples for ours. Non-italic font indicates models based on Gemma-2-27B.
## 1 Introduction
The remarkable advancements in large language models (LLMs) ( DeepSeek-AI, 2024b ; OpenAI, 2025b ) have catalyzed significant shifts in artificial intelligence research, enabling models to perform tasks that require understanding, generation, and nuanced decision-making capabilities. Recently, reinforcement learning (RL) as a post-training method for LLMs has been widely adopted at scale, and resulting in remarkable improvements in human value alignment ( Ouyang et al., 2022 ; Bai et al., 2022a ) , long-term reasoning ( DeepSeek-AI, 2025 ; OpenAI, 2025c ) , and environment adaptation ( OpenAI, 2025a ) for LLMs. Reward modeling (RM) ( Gao et al., 2023 ) , as a crucial component in RL, is essential for generating accurate reward signals for LLM responses. Current studies ( Lightman et al., 2024 ; DeepSeek-AI, 2025 ) also show that, with high-quality and robust rewards in either training or inference time, LLMs can achieve strong performance in specific domains.
However, such high-quality rewards in specific domains are mainly obtained from human-designed environments with clear conditions ( Yao et al., 2022 ; Xie et al., 2024 ) or from hand-crafted rules for verifiable questions, e.g., mathematical problems ( Hendrycks et al., 2021 ; Veeraboina, 2023 ) and coding tasks ( Jimenez et al., 2024 ; Zhuo et al., 2025 ) . In general domains, reward generation is more challenging, as the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. Generalist reward modeling is thus crucial for improving the performance of LLMs in broader applications, either from post-training perspectives, e.g., RL at scale, or from inference perspectives, e.g., RM-guided search. Furthermore, RM performance should be improved by increasing both the training compute ( Gao et al., 2023 ) and the inference compute.
In practice, challenges arise in making RMs both general and effectively scalable in inference time.
The former demands (1) flexibility for different input types and (2) accurate reward generation in various domains. We refer to this paradigm as generalist reward modeling .
Moreover, effective inference-time scalability requires the RM (3) to generate higher-quality reward signals with increased inference compute, and (4) to learn scalable behaviors for better performance-compute scaling.
Existing research on reward modeling demonstrates several paradigms for reward generation, including scalar ( Cobbe et al., 2021 ; Wang et al., 2024e ; Liu et al., 2024 ) , semi-scalar ( Ye et al., 2025a ; Yu et al., 2025b ; Zhang et al., 2025a ) , and generative ( Li et al., 2024a ; Kim et al., 2024 ; Vu et al., 2024 ; Cao et al., 2024 ; Arabzadeh et al., 2024 ; Ye et al., 2025b ; Alexandru et al., 2025 ; Yu et al., 2025a ) approaches, and various scoring patterns, such as pointwise ( Kendall & Smith, 1940 ; Gao et al., 2023 ; Yuan et al., 2024 ; Winata et al., 2025 ; Guo et al., 2025 ) and pairwise ( Park et al., 2024 ; Zheng et al., 2023 ; Jiang et al., 2023 ; Wang et al., 2024c ; Liu et al., 2025 ) . These approaches inherently determine the input flexibility and the inference-time scalability of RMs ( (1)&(3) ), as shown in Figure 2 . For instance, pairwise RMs only consider the relative preference of paired responses, lacking flexibility to accept single or multiple responses as input; scalar RMs could hardly generate diverse reward signals for the same response, which obstructs getting better rewards through sampling-based inference-time scaling methods ( Snell et al., 2025 ) .
Also, different learning methods ( Wang et al., 2024a ; Ankner et al., 2024 ; Wang et al., 2024c ; Mahan et al., 2024 ) have been proposed to improve the quality of rewards, but few of them focus on inference-time scalability and study the interconnection between the learned reward generation behaviors and the effectiveness of inference-time scaling of RMs, resulting in marginal performance improvement ( (2)&(4) ).
Current research ( DeepSeek-AI, 2025 ) indicates that effective inference-time scalability could be enabled by proper learning methods, which raises the question: Can we design a learning method aiming to enable effective inference-time scaling for generalist reward modeling?
In this work, we investigated different approaches for RM, and found that pointwise generative reward modeling (GRM) could unify the scoring of single, paired, and multiple responses within pure language representation, overcoming challenge (1) . We explored that certain principles could guide reward generation within proper criteria for GRMs, improving the quality of rewards, which suggested that inference-time scalability of RM might be achieved by scaling the generation of high-quality principles and accurate critiques . Based on this preliminary, we propose a novel learning method, Self-Principled Critique Tuning (SPCT), to foster effective inference-time scalable behaviors in GRMs. By leveraging rule-based online RL, SPCT enables GRMs to learn to adaptively posit principles and critiques based on the input query and responses, leading to better outcome rewards in general domains (challenge (2) ).
We then come up with DeepSeek-GRM-27B , which is post-trained with SPCT based on Gemma-2-27B ( Team, 2024 ) .
For inference-time scaling, we expand compute usage by sampling multiple times. By sampling in parallel, DeepSeek-GRM could generate different sets of principles and corresponding critiques, and then vote for the final reward. With larger-scale sampling, DeepSeek-GRM could judge more accurately based on more diverse principles, and output rewards with finer granularity , which resolves challenge (3)&(4) .
Furthermore, we train a meta RM in addition to voting for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in multiple comprehensive RM benchmarks without severe domain biases. We also compared the inference-time scaling performance of DeepSeek-GRM-27B with larger models up to 671B parameters, and found it could achieve better performance compared to training-time scaling on model sizes.
Though the current method meets challenges in efficiency and specific tasks,
with efforts beyond SPCT, we believe GRMs with enhanced scalability and efficiency could serve as a versatile interface for generalist reward systems, advancing the frontiers of LLM post-training and inference.
In general, our main contributions are as follows.
1.
We propose a novel approach, Self-Principled Critique Tuning (SPCT), to foster effective inference-time scalability for generalist reward modeling, resulting in DeepSeek-GRM models. And we further introduce a meta RM to effectively improve the inference-time scaling performance of DeepSeek-GRM beyond voting.
2.
We empirically show SPCT significantly improves the quality and inference-time scalability of GRMs over existing methods and several strong public models.
3.
We also applied the SPCT training schedule to LLMs with larger sizes and found that inference-time scaling could outperform model size scaling in training time.
Figure 2: Different paradigms for reward generation, including (a) scalar, (b) semi-scalar, and (c) generative approaches, and different scoring patterns, including (i) pointwise and (ii) pairwise approaches. We list the representative methods for each approach, and corresponding inference-time scalability (whether better rewards could be obtained from multiple sampling) and input flexibility (whether supports rating single and multiple responses).
## 2 Preliminaries
## 2.1 Comparisons of Different RM approaches
As shown in Figure 2 , RM approaches are mainly determined by reward generation paradigms and scoring patterns, which inherently affect the inference-time scalability and the input flexibility of the RM. For reward generation paradigms , we distinguish three main approaches: scalar, semi-scalar, and generative. For scoring patterns , we distinguish two main approaches: pointwise and pairwise.
To expand compute usage in inference time, we focus on sampling-based methods, which generate multiple sets of rewards for the same query and responses, and then aggregate the final reward. Thus, the inference-time scalability of RMs is determined by whether different rewards could be obtained from multiple sampling, where scalar RMs would fail in most cases due to the invariant generation of rewards; and the input flexibility is defined by whether the RM supports rating single, paired, and multiple responses, where pairwise RMs could hardly rate single responses and usually require extra techniques ( Jiang et al., 2023 ; Liu et al., 2025 ) to handle multiple responses.
## Reward Generation Paradigms
Classic RMs adopt the (a) scalar approach to generate rewards ( ℛ \mathcal{R} ), which assigns scalar values to the given query and responses. The scalar approach is further extended to the (b) semi-scalar approach, which generates texts in addition to the scalar value. And the (c) generative approach only generates textual rewards.
ℛ = { 𝑺 (Scalar) ( 𝑺 , 𝑪 ) (Semi-Scalar) 𝑪 (Generative) ∼ r θ ​ ( x , { y i } i = 1 n ) , \mathcal{R}=\begin{cases}\bm{S}&\text{(Scalar)}\\
\left(\bm{S},\bm{C}\right)&\text{(Semi-Scalar)}\\
\bm{C}&\text{(Generative)}\end{cases}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
(1)
where x x is the query, y i y_{i} is the i i -th response, r θ r_{\theta} is the reward function parameterized by θ \theta , 𝑺 ∈ ℝ m , m ≤ n \bm{S}\in\mathbb{R}^{m},m\leq n is the scalar reward, and 𝑪 \bm{C} is the critique.
## Scoring Patterns
We distinguish two main scoring approaches for rewards: pointwise and pairwise. The (i) pointwise approach assigns an individual score to each response:
{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) , ℛ ∼ r θ ​ ( x , { y i } i = 1 n ) , S i ∈ ℝ , \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right),\quad\mathcal{R}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),S_{i}\in\mathbb{R},
(2)
where f point ​ ( ⋅ , ⋅ ) f_{\mathrm{point}}(\cdot,\cdot) is a splitting function.
In contrast, the (ii) pairwise approach can be viewed as a best-of- n n method, selecting a single best response from all candidates:
y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) , ℛ ∼ r θ ​ ( x , { y i } i = 1 n ) , y ^ ∈ { y i } i = 1 n , \hat{y}=f_{\mathrm{pair}}(\mathcal{R},\{y_{i}\}_{i=1}^{n}),\quad\mathcal{R}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\hat{y}\in\{y_{i}\}_{i=1}^{n},
(3)
where f pair ​ ( ⋅ , ⋅ ) f_{\mathrm{pair}}(\cdot,\cdot) is a selection function and n = 2 n=2 in most cases. Though the pairwise approach could be extended to n > 2 n>2 , it could not be applied to single response scoring ( n = 1 n=1 ).
## Representative Methods
Figure 2 illustrates how the three reward generation paradigms (scalar, semi-scalar, generative) can be combined with the two scoring patterns (pointwise, pairwise). Specifically, Bradley-Terry model ( Kendall & Smith, 1940 ) ( (a)+(i) ) is trained with pairwise preference data and outputs scalar rewards pointwisely
{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) = 𝑺 ∈ ℝ n . \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=\bm{S}\in\mathbb{R}^{n}.
(4)
PairRM ( Jiang et al., 2023 ) ( (a)+(ii) ) compares a pair of responses with the sign of the scalar reward
y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) = y ⌊ 1 2 ​ ( 3 − sgn ​ ( 𝑺 ) ) ⌋ , n = 2 , 𝒮 ∈ ℝ . \hat{y}=f_{\mathrm{pair}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=y_{\lfloor\frac{1}{2}(3-\mathrm{sgn}(\bm{S}))\rfloor},\quad n=2,\mathcal{S}\in\mathbb{R}.
(5)
The scalar methods above could barely perform inference-time scaling due to the lack of diversity in reward generation.
CLoud ( Ankner et al., 2024 ) ( (b)+(i) ) generates scalar rewards for each response based on pre-generated critiques, similar to Equation 4 .
LLM-as-a-Judge ( Zheng et al., 2023 ; Wang et al., 2025a ) ( (c)+(ii) ) judges the preference order between paired responses textually,
y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) = y f extract ​ ( 𝑪 ) , n = 2 , \hat{y}=f_{\mathrm{pair}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=y_{f_{\mathrm{extract}}(\bm{C})},\quad n=2,
(6)
where f extract ​ ( ⋅ ) f_{\mathrm{extract}}(\cdot) extracts the index of best response from language representations. However, this approach defaults to neglecting ties of the paired responses. Following Zhang et al. (2025a) , the generation probability of the token that indicates the preference order could be used as the scalar reward ( (b)+(ii) ): 𝒮 = TokenProb ⁡ ( C ^ ) = r θ ​ ( C ^ | x , { y i } i = 1 n ) \mathcal{S}=\mathrm{TokenProb}(\hat{C})=r_{\theta}(\hat{C}|x,\{y_{i}\}_{i=1}^{n}) , where C ^ \hat{C} is a pre-defined token related to the preference order.
However, without additional constraints, GRMs are able to generate pointwise rewards for multiple responses within pure language representations ( (c)+(i) ):
{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) = f extract ​ ( 𝑪 ) , \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=f_{\mathrm{extract}}(\bm{C}),
(7)
where f extract ​ ( ⋅ ) f_{\mathrm{extract}}(\cdot) extracts the rewards assigned to each response from generation results. Usually, the rewards are discrete, and in this work we assign S i ∈ ℕ , 1 ≤ S i ≤ 10 S_{i}\in\mathbb{N},1\leq S_{i}\leq 10 by default. This approach enables both inference-time scalability and input flexibility.
## 2.2 Boosting Reward Quality with Principles
Generalist RM requires generating high-quality rewards beyond specific domains ( Hendrycks et al., 2021 ; Jimenez et al., 2024 ) , where the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. To this end, for general domains, we adopt principles to guide reward generation in place of artificial rules.
Principles for LLMs are first introduced in Constitutional AI ( Bai et al., 2022b ; Sharma et al., 2025 ) , which are hand-crafted criteria that guide the LLMs or curated classifiers to construct safe data pipelines. With principles, the reward generation of GRMs changes to
ℛ = 𝑪 ∼ r θ ​ ( x , { y i } i = 1 n , { p i } i = 1 m ) , \mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i}\}_{i=1}^{m}\right),
(8)
where { p i } i = 1 m \{p_{i}\}_{i=1}^{m} denotes the principles.
We conduct a preliminary experiment to examine the influence of proper principles on reward quality, with the Chat Hard subset of Reward Bench ( Lambert et al., 2024 ) and the IFEval subset of the PPE benchmark ( Frick et al., 2025 ) .
Method
Chat Hard
IFEval
GPT-4o-2024-08-06
76.1
56.0
w/ Self-Gen. Principles
75.9
55.6
w/ Filtered Principles
77.8
57.5
Gemma-2-27B-it
59.1
56.1
w/ Self-Gen. Principles
64.0
55.8
w/ Filtered Principles
68.0
57.3
Table 1: Preliminary experiments on the influence of principles on reward quality. The default setting of DeepSeek-GRM-27B includes self-generated principles.
In the experiment, data samples contain a query and two responses, with the ground-truth label denoting the better response.
We use GPT-4o-2024-08-06 to generate the principles and then pointwise rewards four times for each sample. We filter the principles from the correct reward generation process, where the larger reward value is assigned to the labeled better response. We test different LLMs with principles generated by themselves and the filtered principles, and compare them with the default setting with no principle guidance. The results are shown in Table 1 .
We found that the self-generated principles barely improve performance, but the filtered principles could significantly boost the reward quality. The result is non-trivial and two main conclusions could be drawn:
(a) Current LLMs could generate diverse principles, but not all of them are proper for reward generation. (b) A subset of generated principles could better guide reward generation under correct criteria, indicating a potential of self-bootstrapping.
These findings are the foundation of the usage of online RL to optimize GRMs, where they could learn from the principles generated by themselves with a clear signal to tell whether the principles are proper or not.
Other details are depicted in Appendix D .
Figure 3: Illustration of SPCT, including rejective fine-tuning, rule-based RL, and corresponding scalable behaviors during inference. The inference-time scaling is achieved via naive voting or meta RM guided voting with principles generated at scale, resulting in finer-grained outcome rewards within an expanded value space.
## 3 Self-Principled Critique Tuning (SPCT)
Inspired by the preliminary results, we developed a novel approach for pointwise GRMs to learn generating adaptive and high-quality principles that could effectively guide the generation of critiques, termed Self-Principled Critique Tuning (SPCT). As shown in Figure 3 , SPCT consists of two phases: rejective fine-tuning, as the cold start, and rule-based online RL, reinforcing generalist reward generation by advancing the generated principles and critiques. SPCT fosters these behaviors in GRMs for inference-time scaling as well.
## 3.1 Unpinning Principles from Understanding to Generation
From preliminary experiments in Section 2.2 , we found that proper principles could guide reward generation within certain criteria, which is critical for high-quality rewards. However, it remains challenging to generate effective principles for generalist RM at scale. To address this challenge, we propose to unpin principles from understanding to generation, i.e. view principles as a part of reward generation instead of a preprocessing step.
Formally, principles guide the generation of rewards following Equation 8 , when principles are pre-defined. GRMs could generate principles themselves, and then generate critiques based on the principles, formalized as
{ p i } i = 1 m ∼ p θ ​ ( x , { y i } i = 1 n ) , ℛ = 𝑪 ∼ r θ ​ ( x , { y i } i = 1 n , { p i } i = 1 m ) , \{p_{i}\}_{i=1}^{m}\sim p_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\quad\mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i}\}_{i=1}^{m}\right),
(9)
where p θ p_{\theta} is the principle generation function parameterized by θ \theta , that shares the same model with reward generation r θ r_{\theta} . In practice, they are implemented with the same language head in LLMs.
This shift enables principles to be generated based on the input query and responses, adaptively aligning reward generation process, and the quality and granularity of the principles and corresponding critiques could be further improved with post-training on GRMs.
With the principles generated at scale , GRMs could potentially output rewards with finer granularity and broader consideration to achieve better inference-time scalability.
## 3.2 Rule-Based Reinforcement Learning
To optimize principle and critique generation in GRMs simultaneously, we propose SPCT, which integrates rejective fine-tuning and rule-based RL . The former serves as a cold start.
Rejective Fine-Tuning (Cold Start)
The core idea of the rejective fine-tuning stage is to train the GRM to generate principles and critiques in the correct format and for various input types. Unlike previous works ( Vu et al., 2024 ; Cao et al., 2024 ; Alexandru et al., 2025 ) that mix RM data for single, paired, and multiple responses in different formats, we adopt pointwise GRM, introduced in Section 2.1 , to flexibly generate rewards for any number of responses in the same format. For data construction, besides general instruction data, we sample trajectories with a pretrained GRM by giving the query and corresponding responses. Each RM data point contains a query and one or multiple responses to the query, as well as the ground-truth label denoting the best response. For each RM data point, the sampling of principles and critiques is performed N RFT N_{\mathrm{RFT}} times. The rejection strategy is also unified, which is to reject trajectories with predicted rewards that are incorrect, and the query and responses with all N RFT N_{\mathrm{RFT}} trajectories correct (too easy).
Formally, let r i r_{i} denote the ground-truth reward for the i i -th response y i y_{i} to the query x x , the predicted pointwise rewards { S i } i = 1 n \{S_{i}\}_{i=1}^{n} are correct if
{ ∀ i ≠ j , S j > S i , j = arg ​ max l { r l } l = 1 n , if ​ n ≥ 2 , S 1 = r 1 , if ​ n = 1 . \begin{cases}\forall i\neq j,\quad S_{j}>S_{i},\quad j=\argmax_{l}\{r_{l}\}_{l=1}^{n},&\text{if }n\geq 2,\\
S_{1}=r_{1},&\text{if }n=1.\end{cases}
(10)
with a guarantee that the ground-truth rewards only contain one maximum. However, similar to previous works ( Zhang et al., 2025a ) , we found pretrained GRMs could hardly generate correct rewards for a portion of queries and corresponding responses within limited sampling quota. Thus, we optionally append arg ​ max l ⁡ { r l } l = 1 n \argmax_{l}\{r_{l}\}_{l=1}^{n} to the prompt of the GRM, termed hinted sampling , with the expectation that the predicted rewards to align with the ground truth, besides non-hinted sampling . Specifically, an additional segment “The best response is: Response arg ​ max l ⁡ { r l } l = 1 n \argmax_{l}\{r_{l}\}_{l=1}^{n} ” will be appended to the input. For hinted sampling, each query and the corresponding responses are sampled once, and trajectories are only rejected when incorrect. Beyond previous studies ( Li et al., 2024a ; Mahan et al., 2024 ) , we observed that hinted sampled trajectories sometimes take shortcuts in the generated critique, especially for reasoning tasks, indicating the necessity and potential benefits of online RL for the GRM.
Rule-Based RL The GRM is further fine-tuned with rule-based online RL. Specifically, we use the original setting of GRPO ( Shao et al., 2024 ) with rule-based outcome rewards. During rollout, the GRM generates principles and critiques based on the input query and responses, and then the predicted reward is extracted and compared to the ground truth with accuracy rules. Unlike DeepSeek-AI (2025) , no format rewards are used. Instead, a larger coefficient for the KL penalty is applied to ensure the format and avoid severe biases. Formally, the reward for the i i -th output o i o_{i} to the given query x x and responses { y i } i = 1 n \{y_{i}\}_{i=1}^{n} is
r ^ i = { 1 , if n ≥ 2 and ∀ i ′ ≠ j ′ , S j ′ > S i ′ , j ′ = arg ​ max l { r l } l = 1 n , 1 , if ​ n = 1 ​ and ​ S 1 = r 1 , − 1 , otherwise , \hat{r}_{i}=\begin{cases}1,&\text{if }n\geq 2\text{ and }\forall i^{\prime}\neq j^{\prime},\quad S_{j^{\prime}}>S_{i^{\prime}},\quad j^{\prime}=\argmax_{l}\{r_{l}\}_{l=1}^{n},\\
1,&\text{if }n=1\text{ and }S_{1}=r_{1},\\
-1,&\text{otherwise},\end{cases}
(11)
where the pointwise rewards { S i } i = 1 n \{S_{i}\}_{i=1}^{n} are extracted from o i o_{i} . The reward function encourages GRMs to distinguish the best responses with online optimized principles and critiques, in favor of effective inference-time scaling. The reward signal could be obtained seamlessly from any preference dataset and labeled LLM responses.
## 4 Inference-Time Scaling with SPCT
To further improve the performance of DeepSeek-GRM for generalist reward generation using more inference compute, we explore sampling-based strategies to achieve effective inference-time scalability.
Voting with Generated Rewards
Voting is a widely adopted method for inference-time scaling in RM. Recalling the approaches in Section 2.1 , we demonstrate voting results of k k samples for semi-scalar and generative RMs. For semi-scalar RMs ( Ankner et al., 2024 ; Zhang et al., 2025a ) , voting is performed as averaging:
𝑺 ∗ = 1 k ∑ i = 1 k 𝑺 i , { ℛ i = ( 𝑺 i , 𝑪 i ) } i = 1 k ∼ r θ ( x , { y i } i = 1 n ) , ~\bm{S}^{*}=\frac{1}{k}\sum_{i=1}^{k}\bm{S}_{i},\quad\{\mathcal{R}_{i}=(\bm{S}_{i},\bm{C}_{i})\}_{i=1}^{k}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
(12)
where 𝑺 ∗ \bm{S}^{*} is the final reward. In practice, the scalar value has limited variance which could hinder the scalability.
For pairwise GRMs ( Mahan et al., 2024 ; Wang et al., 2024c ) , voting is performed as selecting the response identified to be the best with the highest frequency, i.e. majority:
y ^ ∗ = arg ​ max y ∑ i = 1 k 𝕀 ( y = y ^ i ) , { ℛ i = 𝐂 i } i = 1 k ∼ r θ ( x , { y i } i = 1 n ) , \hat{y}^{*}=\argmax_{y}\sum_{i=1}^{k}\mathbb{I}(y=\hat{y}_{i}),\quad\{\mathcal{R}_{i}=\bm{C}_{i}\}_{i=1}^{k}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
(13)
where y ^ ∗ \hat{y}^{*} is the final predicted best response, f pair ​ ( ⋅ , ⋅ ) f_{\mathrm{pair}}(\cdot,\cdot) is a selection function, y ^ i = f pair ​ ( 𝑪 i , { y i } i = 1 n ) \hat{y}_{i}=f_{\mathrm{pair}}(\bm{C}_{i},\{y_{i}\}_{i=1}^{n}) is the individually selected best response of each sample, and 𝕀 ⁡ ( ⋅ ) \mathbb{I}(\cdot) is the indicator function. Though the voting process is scalable, the majority voted result might be biased since ties are not allowed in each sample, and may not be able to tell apart subtle differences between responses due to the lack of quantitative scores.
The voting process for pointwise GRMs is defined as summing the rewards:
S i ∗ = ∑ j = 1 k S i , j , { p i , j } i = 1 m j ∼ p θ ( x , { y i } i = 1 n ) , ℛ j = 𝑪 j ∼ r θ ( x , { y i } i = 1 n , { p i , j } i = 1 m j ) , j = 1 , … , k , \footnotesize S_{i}^{*}=\sum_{j=1}^{k}S_{i,j},\hskip 8.50012pt\{p_{i,j}\}_{i=1}^{m_{j}}\sim p_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\mathcal{R}_{j}=\bm{C}_{j}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i,j}\}_{i=1}^{m_{j}}\right),j=1,...,k,
(14)
where S i ∗ S_{i}^{*} is the final reward for the i i -th response ( i = 1 , … , n i=1,...,n ) and { S i , j } i = 1 n = f point ​ ( 𝑪 j , { y i } i = 1 n ) \{S_{i,j}\}_{i=1}^{n}=f_{\mathrm{point}}(\bm{C}_{j},\{y_{i}\}_{i=1}^{n}) is the j j -th set of pointwise rewards. Since S i , j S_{i,j} is usually set within a small discrete range, e.g., { 1 , … , 10 } \{1,...,10\} , the voting process actually expands the reward space by k k times, and enables the GRM to generate a large number of principles, which benefits the quality and granularity of the final rewards . An intuitive explanation is that, if each principle could be viewed as a proxy of judgement perspectives, a larger number of principles may reflect the real distribution more accurately, leading to scaling effectiveness. Notably, to avoid positional biases and for diversity, responses are shuffled before sampling.
Meta Reward Modeling Guided Voting
The voting process of DeepSeek-GRM requires multiple sampling and a few generated principles and critiques might be biased or low-quality due to randomness or model limitations. Thus, we train a meta RM to guide the voting process. The meta RM is a pointwise scalar RM, trained to identify the correctness of the principles and critiques generated by DeepSeek-GRM, with the binary cross-entropy loss, where the label is identified based on Equation 10 . The prompt template is in Appendix G , integrates the query, candidate responses, corresponding principles, and the critiques.
The dataset comprises trajectories from non-hinted sampling in the RFT stage, and also trajectories sampled from the DeepSeek-GRM to be guided, to both provide enough positive and negative rewards and alleviate the gap between training and inference policy as suggested by Chow et al. (2025) . The guided voting is simple: The meta RM outputs meta rewards for k k sampled rewards, and the final outcome is voted by rewards with top k meta ≤ k k_{\mathrm{meta}}\leq k meta rewards, so that filtering out low-quality samples.
## 5 Results on Reward Modeling Benchmarks
Model
RB
PPE Pref.
PPE Correct.
RMB
Overall
Avg. Rank ( ↓ \downarrow )
Reported Results of Public Models
Skywork-Reward-Gemma-2-27B
94.1
56.6
56.6
60.2
66.9
-
DeepSeek-V2.5-0905
81.5
62.8
58.5
65.7
67.1
-
Gemini-1.5-Pro
86.8
66.1
59.8
56.5
67.3
-
ArmoRM-8B-v0.1
90.4
60.6
61.2
64.6
69.2
-
InternLM2-20B-Reward
90.2
61.0
63.0
62.9
69.3
-
LLaMA-3.1-70b-Instruct
84.1
65.3
59.2
68.9
69.4
-
Claude-3.5-sonnet
84.2
65.3
58.8
70.6
69.7
-
Nemotron-4-340B-Reward
92.0
59.3
60.8
69.9
70.5
-
GPT-4o
86.7
67.1
57.6
73.8
71.3
-
Reproduced Results of Baseline Methods
LLM-as-a-Judge
83.4
64.2
58.8
64.8
67.8
4.50
DeepSeek-BTRM-27B
81.7
68.3
66.7
57.9
68.6
3.50
CLoud-Gemma-2-27B
82.0
67.1
62.4
63.4
68.7
3.50
DeepSeek-PairRM-27B
87.1
65.8
64.8
58.2
69.0
2.75
Results of Our Method
DeepSeek-GRM-27B-RFT (Ours)
84.5
64.1
59.6
67.0
68.8
4.00
DeepSeek-GRM-27B (Ours)
86.0
64.7
59.8
69.0
69.9
2.75
Results of Inference-Time Scaling (Voting@32)
DeepSeek-GRM-27B (Ours)
88.5
65.3
60.4
69.7
71.0
-
DeepSeek-GRM-27B (MetaRM) (Ours)
90.4
67.2
63.2
70.3
72.8
-
Table 2: Overall results of different methods and models on RM benchmarks. Underlined numbers indicate the best performance, bold numbers indicate the best performance among baseline and our methods, and italicized font denotes scalar or semi-scalar RMs. For meta RM guided voting (MetaRM), k meta = 1 2 ​ k k_{\mathrm{meta}}=\frac{1}{2}k . “Avg. Rank ( ↓ \downarrow )” is calculated based on the rank (from 1 to 6, better ranks denoted by darker shades) of baseline and our methods on each dataset.
## 5.1 Experiment Settings
Benchmarks and Evaluation Metrics
We evaluate the performance of different methods on various RM benchmarks of different domains: Reward Bench (RB) ( Lambert et al., 2024 ) , PPE (Preference and Correctness subsets) ( Frick et al., 2025 ) , RMB ( Zhou et al., 2025 ) , ReaLMistake ( Kamoi et al., 2024 ) . We use the standard evaluation metrics for each benchmark: accuracy of picking the best response from a set of responses in Reward Bench, PPE, and RMB, and ROC-AUC for ReaLMistake. To deal with ties of the predicted rewards for multiple responses, we shuffle the responses and determine the best response by arg ​ max i ⁡ S i \argmax_{i}S_{i} , where S i S_{i} is the predicted reward for the i i -th response after shuffling. Details are in Appendix D .
Method Implementation
For the baseline methods, we re-implement LLM-as-a-Judge ( Zheng et al., 2023 ) , DeepSeek-BTRM-27B (Bradley-Terry model) ( Kendall & Smith, 1940 ) , CLoud-Gemma-2-27B ( Ankner et al., 2024 ) , and DeepSeek-PairRM-27B ( Jiang et al., 2023 ) based on Gemma-2-27B ( Team, 2024 ) and with all compatible training data and settings as DeepSeek-GRM. For our methods, we implement DeepSeek-GRM-27B-RFT based on Gemma-2-27B, and DeepSeek-GRM on different sizes of LLMs, including DeepSeek-V2-Lite (16B MoE) ( DeepSeek-AI, 2024a ) , Gemma-2-27B, DeepSeek-V2.5 (236B MoE), and DeepSeek-V3 (671B MoE) ( DeepSeek-AI, 2024b ) . The meta RM is trained on Gemma-2-27B.
Default results are reported with greedy decoding, and the inference-time scaling uses temperature = 0.5 \mathrm{temperature}=0.5 .
Other details are provided in Appendix C .
## 5.2 Results and Analysis
Performance on RM Benchmarks
The overall results of different methods and models on RM benchmarks are shown in Table 2 . We compare the performance of DeepSeek-GRM-27B with the reported results of public models and the reproduced results of baseline methods. We find that DeepSeek-GRM-27B outperforms the baseline methods in overall performance, and achieves competitive performance with strong public RMs, such as Nemotron-4-340B-Reward and GPT-4o; with inference-time scaling, DeepSeek-GRM-27B could further improve and achieve the best overall results. For detailed comparisons, scalar (DeepSeek-BTRM-27B) and semi-scalar (CLoud-Gemma-2-27B) RMs demonstrate biased results on different benchmarks, with significantly better performance on verifiable tasks (PPE Correctness) than all generative RMs, but fail in different other benchmarks, respectively. Nonetheless, most public scalar RMs also exhibit severe domain biases. The PairRM approach could alleviate the problem. LLM-as-a-Judge shows similar trends with DeepSeek-GRM-27B with lower performance, potentially due to the lack of training on rating single responses. In conclusion, SPCT improves the generalist reward generation capability of GRMs, with significantly fewer biases compared to scalar and semi-scalar RMs .
Model
Overall
Reported Results of Public Models
Nemotron-4-340B-Reward
70.5
GPT-4o
71.3
Results of Inference-Time Scaling (Voting@1)
LLM-as-a-Judge
67.0
CLoud-Gemma-2-27B
68.5
DeepSeek-GRM-27B-RFT (Ours)
67.8
DeepSeek-GRM-27B (Ours)
67.9
Results of Inference-Time Scaling (Voting@8)
LLM-as-a-Judge
67.6 ( +0.6 )
LLM-as-a-Judge w/ TokenProb
68.1 ( +1.1 )
CLoud-Gemma-2-27B
68.8 ( +0.3 )
DeepSeek-GRM-27B-RFT (Ours)
69.3 ( +1.5 )
DeepSeek-GRM-27B (Ours)
70.6 ( +2.7 )
DeepSeek-GRM-27B (MetaRM) (Ours)
72.0 ( +4.1 )
Results of Further Inference-Time Scaling (Voting@32)
DeepSeek-GRM-27B (Ours)
71.0 ( +3.1 )
DeepSeek-GRM-27B (MetaRM) (Ours)
72.8 ( +4.9 )
Table 3: Inference-time scalability results of different methods on RM benchmarks. Settings are the same as Table 2 .
Method
Overall
Results of Greedy Decoding
DeepSeek-GRM-27B
69.9
w/o Principle Generation
67.5
w/o Rejective Sampling
68.7
DeepSeek-GRM-27B-RFT
68.8
w/o Hinted Sampling (①)
68.0
w/o Non-Hinted Sampling (②)
67.4
w/o Rejective Sampling (①&②)
66.1
w/o General Instruction Data
63.3
Results of Inference-Time Scaling (Voting@8)
DeepSeek-GRM-27B
70.6
w/o Principle Generation
68.0
Results of Inference-Time Scaling (Voting@32)
DeepSeek-GRM-27B
71.0
DeepSeek-GRM-27B ( k meta = 1 k_{\mathrm{meta}}=1 )
71.5
DeepSeek-GRM-27B ( k meta = 8 k_{\mathrm{meta}}=8 )
72.7
DeepSeek-GRM-27B ( k meta = 16 k_{\mathrm{meta}}=16 )
72.8
Table 4: Ablation studies for different components of the proposed SPCT. Bold numbers indicate the best performance.
Inference-Time Scalability
The inference-time scaling results of different methods are shown in Table 4 , and the overall trends are demonstrated in Figure 1 . Details are in Appendix D.3 . With up to 8 samples, we find that DeepSeek-GRM-27B has the highest performance increase over the greedy decoding and sampling results. DeepSeek-GRM-27B further shows a strong potential to increase performance with larger inference compute, up to 32 samples. We attribute the effectiveness to the refined principle generation, which expands the output length in a structural way and guides the outcome rewards closer to the ground-truth distribution.
The meta RM also reveals its validity in filtering low-quality trajectories for DeepSeek-GRM on each benchmark.
Voted with token probabilities, LLM-as-a-Judge also shows a significant performance increase, indicating that the token probability as quantitative weights could help the reliability over mere majority voting on discrete indices. For CLoud-Gemma-2-27B, the performance increase is limited, mainly due to the lack of variance in scalar reward generation, even though the critique has changed a lot. In summary, SPCT improves the inference-time scalability of GRMs, and the meta RM further boosts the scaling performance in general .
Ablation Study
Table 4 shows the ablation study results of different components of the proposed SPCT, detailed results are listed in Appendix D.3 . Surprisingly, without the cold start with rejective sampled critique data, general-instruction-tuned GRMs still improve significantly after undergoing the online RL ( 66.1 → 68.7 66.1\rightarrow 68.7 ). Also, the non-hinted sampling seems more important than the hinted sampling, potentially because of the shortcuts appearing in hinted sampled trajectories. These indicate the importance of online training for GRMs . Aligned with previous works ( Cao et al., 2024 ) , we confirm that the general instruction data is essential for the performance of GRMs. We find that the principle generation is crucial for the performance of both greedy decoding and inference-time scaling of DeepSeek-GRM-27B . For inference-time scaling, the meta RM guided voting shows robustness with different k meta k_{\mathrm{meta}} . Further analysis on the generalist RM performance, including input flexibility, domain generalization of training data, etc., is discussed in Appendix E .
Scaling Inference and Training Costs
We further investigate the inference-time and training-time scaling performance of DeepSeek-GRM-27B, by post-training LLMs of different sizes. The models are tested on the Reward Bench, and the results are shown in Figure 4 .
We find that direct voting with 32 samples of DeepSeek-GRM-27B could achieve comparable performance to the 671B MoE model, and the meta RM guided voting could achieve the best results with 8 samples, demonstrating the effectiveness of inference-time scaling of DeepSeek-GRM-27B compared to scaling model sizes .
Moreover, we test DeepSeek-R1-0120 with a downsampled test set containing 300 samples, and find that its performance is even worse than the 236B MoE RFT model, indicating that expanding long chain-of-thoughts for reasoning tasks could not significantly improve the performance of generalist RM.
(a) Inference-time scaling results of DeepSeek-GRM-16B and DeepSeek-GRM-27B ( k meta = 1 2 ​ k k_{\mathrm{meta}}=\frac{1}{2}k ).
(b) Training-time scaling results with different model sizes, using greedy decoding except DeepSeek-R1.
Figure 4: Inference-time scaling performance v.s. training-time scaling performance on the Reward Bench benchmark.
## 6 Related Work
Generative Reward Models
GRMs represent a paradigm shift from scalar RMs ( Ouyang et al., 2022 ) , modeling reward as textual feedback or scores. ( Li et al., 2024a ; Kim et al., 2024 ; Wang et al., 2024c ; Cao et al., 2024 ; Vu et al., 2024 ; Alexandru et al., 2025 ) , enabling richer reward representations and more flexible for judging single and multiple responses. Previously, the LLM-as-a-judge method ( Zheng et al., 2023 ; Wang et al., 2025a ) accommodates reference-based or reference-free pairwise judgement for evaluating LLMs. Recent studies use offline and online RL to train GRMs ( Wu et al., 2024 ; Mahan et al., 2024 ; Yu et al., 2025a ; Ye et al., 2025b ; Chen et al., 2025 ) , incorporate tools and external knowledge with GRMs ( Li et al., 2024b ; Peng et al., 2025 ) , and even train GRMs as an interface to adjust rewards from environments ( Baker et al., 2025 ) .
Though these methods face challenges in efficiency, they demonstrate the potential to improve rewards at scale, towards a more generalist reward system.
Inference-Time Scaling for LLMs
Inference-time scaling for LLMs has been a critical research direction parallel with scaling LLMs in training time. Studies focus on sampling and RM guided aggregation ( Lightman et al., 2024 ; Brown et al., 2024 ; Snell et al., 2025 ; Wu et al., 2025 ) . Recently, long-horizon chain-of-thoughts ( Wei et al., 2022 ) incentivized from LLMs significantly improve the reasoning capabilities of the models in both solving ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; OpenAI, 2025c ) and judging ( Zheng et al., 2025 ; Kim et al., 2025 ) difficult verifiable questions, as another format of inference-time scaling. However, we do not find an effective way to incentivize long-horizon reward generation for generalist reward modeling as DeepSeek-AI (2025) , and we leave the combination of reasoning and principle-guided reward generation for future engineering efforts.
There is also research using scalable rewards or verifiers to improve the performance of policy models, in domains of coding ( Chen et al., 2023 ) , reasoning ( Lifshitz et al., 2025 ) , etc. Thus, the development of inference-time scalable generalist RMs in this work might also contribute to the general performance of policy models by inference-time co-scaling.
## 7 Conclusion and Future Work
We introduced Self-Principled Critique Tuning (SPCT), a method that enhances the scalability of inference time for generalist reward modeling. With rule-based online RL, SPCT enables adaptive generation of principles and critiques, significantly boosting reward quality and inference-time scalability for GRMs in diverse domains. Empirical results demonstrate that DeepSeek-GRM surpasses baseline methods and a few strong public RMs, and shows notable improvement through inference-time scaling, particularly with the guidance of the meta RM.
Future directions could include integrating GRMs into online RL pipelines as versatile interfaces of reward systems, exploring inference-time co-scaling with policy models, or serving as robust offline evaluators for foundation models.
## Ethics Statement
Our proposed method, Self-Principled Critique Tuning (SPCT), aims to enhance inference-time scalability of generative reward models (GRMs) for general domains. While this advancement promotes accuracy and consistency in reward modeling, several ethical implications might warrant explicit consideration.
Firstly, even though through our empirical analysis that DeepSeek-GRM shows fewer biases on different domains, the automated generation of principles and critiques can inadvertently perpetuate or amplify biases when the training data is toxic. We argue that further investigation in the meta RM and other bias mitigation strategies should be prioritized to ensure equitable outcomes.
Also, our approach does not aim to diminish human oversight. Instead, we advocate maintaining human-in-the-loop frameworks, and developing reliable proxy methods, like SPCT, to scale human oversight more efficiently and effectively.
Secondly, expanded applicability of the inference-time scalable GRMs across diverse domains might raise concerns regarding transparency, accountability, etc. Since the reward generation behavior is largely emerged from self-bootstrapping, the potential for unfaithful principles and critiques is non-negligible. We demonstrate case studies in Appendix F.1 and limitations in Appendix B , and open-source the model under public supervision, which is essential for maintaining trust and ensuring responsible deployment of the artifact.
Finally, robust validation and ongoing vigilance across varied RM benchmarks and practical scenarios remain crucial. Ethical use of DeepSeek-GRM necessitates proactive management of risks and continuous evaluation against biases, requiring efforts in research about RM evaluation.

=== BODY (tail) ===