Process Reward Models: PRM Training, Math Reasoning

Article links

Make inline references clickable

Language models learn to generate text, but they do not inherently learn to reason well. When you ask a model to solve a multi-step math problem, write a proof, or debug a program, the model's objective during training is simply to predict the next token correctly. Nothing in that objective forces the model to check whether intermediate steps are logically valid. A model that produces a plausible-looking chain of reasoning can still arrive at a wrong answer through a sequence of subtle errors.

The challenge is producing correct final answers and correct reasoning. To train models that reason reliably, we need a way to evaluate whether the final answer is right and whether each step along the way was valid. This is the core motivation for process reward models.

A process reward model (PRM) is a model that assigns a score to individual steps in a reasoning chain, rather than evaluating only the final outcome. Instead of asking "did the model get the right answer?", a PRM asks "was this step logically sound?". This distinction matters because it enables training and search procedures that optimize the quality of reasoning at every stage, not just at the end.

Process reward models connect reinforcement learning from human feedback (RLHF) and structured reasoning. They extend the idea of a reward model, which you encountered in earlier chapters on RLHF, to the sequential, structured setting of multi-step problem solving. To understand why this extension is necessary, it helps to trace the history of the problem and appreciate exactly where simpler reward formulations break down.

Before diving into the mechanics of process reward models, it is worth spending time on the underlying problem they solve. Language models are trained to minimize prediction loss over large corpora of text. This objective works extraordinarily well for tasks that are essentially about pattern completion: language generation, translation, summarization, question answering from context. But it has a fundamental misalignment with tasks that require reliable, multi-step deductive reasoning.

Consider what happens when a large language model is asked to solve a competition-level algebra problem. During pretraining, the model has seen many algebraic manipulations. It has learned which symbols tend to appear together, which operations follow which others, and what a typical solution looks like. When it generates a response, it draws on these distributional patterns. The result often looks like a valid proof, uses correct notation, and arrives at a numerical answer. But the model has no internal representation of "logical necessity." It is generating tokens that have high probability given the context, and correct algebraic manipulations have high probability because they appeared frequently in the training data.

This distinction between distributional plausibility and logical validity becomes critical as problems become harder. For routine algebra, distributional plausibility and logical validity largely coincide, because the training distribution contains many examples of the same problem type. For novel or difficult problems, the correlation breaks down. The model can generate reasoning that looks fluent and structured but contains subtle errors: an incorrect application of a theorem, an invalid step in a derivation, a false assumption carried through multiple steps.

The research community recognized this mismatch early in the scaling era. Papers on chain-of-thought prompting showed that asking models to generate intermediate reasoning steps, rather than jumping directly to answers, substantially improved performance on complex tasks. But chain-of-thought prompting does not guarantee that those steps are correct. It simply invites the model to generate token sequences that resemble correct reasoning, which is much easier to fake than to achieve.

The question becomes: how do you train a model to reason correctly, not just to generate plausible-looking reasoning? This requires a training signal that distinguishes correct steps from incorrect ones, rather than only distinguishing correct final answers from incorrect ones. Process reward models provide exactly this signal.

The conceptual lineage of process reward models runs through reinforcement learning from human feedback. RLHF, as you have seen in prior chapters, trains a reward model to predict human preference between responses and then uses reinforcement learning to fine-tune the policy toward higher-scoring outputs. This approach works well when the quality of a response can be assessed as a whole: which of these two summaries is better? Which response is more helpful?

But for reasoning tasks, whole-response assessment misses the critical distinction between good reasoning and lucky reasoning. A response that gets the right answer via a flawed proof deserves a different reward than one that gets the right answer via a correct proof. Outcome-based RLHF cannot make this distinction; it rewards both equally if the final answer is correct.

The natural generalization is to assess reasoning at the level of individual steps. This is what process supervision does. The term "process supervision" was formally introduced and analyzed in work by Lightman et al. at OpenAI in 2023, though the underlying idea had appeared in various forms in earlier work on reward shaping and potential-based reward functions in reinforcement learning.

The Lightman et al. paper made the comparison concrete: they trained both outcome-supervised and process-supervised reward models on the MATH benchmark (a collection of competition-level mathematics problems) and showed that process supervision substantially outperformed outcome supervision, especially at higher difficulty levels where reasoning chains are longer and more complex. This empirical validation catalyzed significant research attention toward PRMs.

We will explore how they differ from simpler reward formulations, how they are trained, where they work best, and where they fall short.

To understand why process reward models are useful, start with the simpler approach: outcome reward models (ORMs).

An outcome reward model evaluates a complete solution and assigns a single score based on whether the final answer is correct. Given a problem and a full response, the ORM produces a scalar signal indicating solution quality. This approach is natural, simple to implement, and works well when you have access to ground-truth final answers.

The appeal of ORMs is their simplicity. If you can automatically verify the final answer (for example, by checking whether a mathematical answer matches a known solution, or whether generated code passes a test suite), you can generate a training signal without human annotation. This is how many early RLHF pipelines worked: generate a solution, check the answer, assign a reward, update the policy.

The problem with outcome reward is that it is sparse and delayed. Consider a 10-step mathematical derivation. If step 3 contains a subtle algebraic error that propagates through all subsequent steps, the model receives no signal about where the reasoning went wrong. It only knows the final answer was incorrect. This is analogous to telling a student "your answer is wrong" without any feedback on which part of their reasoning was flawed. The learning signal is weak because it is not connected to the individual decisions that caused the error.

Sparsity has a direct effect on sample efficiency. In reinforcement learning, sparse rewards are notoriously difficult to optimize because the signal is so infrequent and so far removed from the individual actions that produced it. Techniques like reward shaping, potential-based reward functions, and hierarchical RL exist precisely to address this problem by introducing intermediate reward signals. Process reward models are, in essence, a domain-specific form of reward shaping for multi-step reasoning.

Process reward models provide a dense reward signal instead. Each step in the chain receives its own score. If step 3 contains an error, the PRM assigns a low score to step 3, regardless of whether subsequent steps recover or the final answer happens to be correct by coincidence. This fine-grained feedback allows training procedures to distinguish between "right answer via wrong reasoning" and "right answer via correct reasoning."

The difference between outcome and process reward becomes particularly important in longer reasoning chains. In a 20-step derivation, the probability that all steps are correct given only a correct final answer is not high: the model might have made compensating errors, used a non-generalizable shortcut, or gotten lucky with an algebraic cancellation. Outcome reward cannot distinguish any of these from sound reasoning. Process reward can.

Credit assignment is the question of which actions deserve credit for an outcome. In sequential decision-making, this is one of the central challenges: when a sequence of decisions leads to a reward, how do you know which decisions mattered?

Temporal difference learning and Monte Carlo methods in reinforcement learning both address credit assignment through different means: TD methods propagate rewards backward through time using bootstrapped value estimates, while Monte Carlo methods average returns over entire trajectories. Both approaches work when rewards are reasonably frequent, but they degrade substantially as the gap between actions and rewards grows.

With outcome reward applied to long reasoning chains, credit assignment is left entirely to the learning algorithm. The optimizer must infer, from the signal "this solution was wrong," which of the many steps in the reasoning chain contributed to the failure. In a chain of 15 steps, this requires attributing the outcome correctly across a very large space of possible step combinations. The gradient signal becomes increasingly diluted, and the optimizer may converge to policies that avoid certain patterns for the wrong reasons or that produce superficially correct-looking reasoning without improving correctness.

Process reward solves credit assignment explicitly by labeling each step. Step 7 receives a high score if step 7 is correct, independent of what happens in steps 8 through 20. The credit assignment problem is reduced from a complex inference problem to a direct annotation: was this step valid? The optimizer can now learn, with a strong gradient signal, exactly which step patterns are associated with correctness.

This explicit credit assignment also has implications for interpretability. When a PRM assigns a low score to step 7 in a 15-step chain, you know step 7 is the problem. This makes PRM-based feedback substantially more useful as a debugging tool than ORM-based feedback, which only tells you that something went wrong somewhere in the chain.

There are specific patterns where outcome reward models fail systematically:

Coincidental correctness. A model might reach the correct final answer through erroneous intermediate steps that happen to cancel out. This is common in algebra, where two errors can offset each other. An ORM rewards this behavior, effectively training the model to use flawed reasoning that occasionally produces correct answers.

Reward hacking. Models trained with outcome reward sometimes discover shortcuts that satisfy the scoring criteria without developing reasoning ability. For example, a model might learn to produce a final numerical answer in the correct format and range without solving the problem, if the scoring function is not sufficiently discriminating.

Long-horizon degradation. As reasoning chains grow longer, the probability that a single error in an early step propagates and causes a wrong final answer increases. The outcome reward becomes harder to use as a learning signal because the signal is noisy: many correct paths share the label "wrong answer" simply because of one early mistake, and the optimizer receives contradictory feedback. The same step can receive positive reward in one trajectory (where it led to a correct final answer) and negative reward in another (where it led to an incorrect one), not because the step itself was different, but because later steps varied. Process reward eliminates this noise by evaluating each step independently.

Distributional shift during training. As the policy improves and generates different distributions of reasoning chains, the outcome reward signal shifts accordingly. Early in training, a step that was usually followed by correct reasoning might later appear in chains that have drifted toward different patterns. The outcome reward provides no signal to stabilize this: it only sees the final answer. A process reward model, updated to match the current policy's distribution, provides a more stable and consistent learning signal.

A process reward model is a neural network that takes as input a problem and a partial or complete reasoning chain and produces a scalar score for the most recent step. The architecture is typically a transformer-based language model fine-tuned to perform step-level evaluation. Understanding both the architectural choices and the training procedure is essential for appreciating both the strengths and the limitations of current PRMs.

The input to a PRM consists of the original problem statement, all preceding steps in the reasoning chain, and the step being evaluated. The model must use context from earlier steps to assess whether the current step is valid. For example, if a step claims "therefore ," the PRM needs to know what was established in prior steps to judge whether that claim follows.

A common input format represents each step as a block of text, with steps separated by a special delimiter token. The PRM produces a prediction at each delimiter position, yielding one score per step. This framing turns the PRM into a token-level classification problem: at each step boundary, predict whether the step is correct.

The use of a full language model as the backbone is important for two reasons. First, step correctness is inherently a language understanding problem: you need to understand the semantic content of each step and compare it to the problem context and prior steps. Second, the scoring model must generalize across many problem types, mathematical domains, and notation conventions. A transformer trained on mathematical text develops the representational capacity to handle this diversity.

In the PRM800K setup from Lightman et al., the backbone was a GPT-4 class model fine-tuned specifically for step evaluation. The input was formatted as a sequence of problem text followed by steps, with a special token inserted between steps. The model predicted a score at each step-boundary token, effectively running a per-step binary classifier on top of the language model's contextualized representations.

One design choice is whether to score each step in isolation (re-running the model from scratch for each step) or to score all steps in a single forward pass. The single-pass approach is more efficient: one inference call processes the entire chain and outputs one score per step boundary. The isolation approach might seem to provide cleaner credit assignment, but in practice the contextual dependencies between steps matter for correct evaluation, so single-pass scoring is both more efficient and more accurate.

Training a PRM requires step-level labels: for each step in each solution, a binary or continuous label indicating whether the step is correct. Obtaining these labels is the core challenge of PRM training.

There are two main approaches to generating step-level labels:

Human annotation. Human raters read each step in a reasoning chain and judge whether it is mathematically or logically valid. This produces high-quality labels but is expensive and difficult to scale. The Process Reward Model dataset from Lightman et al. (OpenAI, 2023) used this approach, collecting step-level labels from human mathematicians for solutions to MATH benchmark problems. Human annotators were asked to label each step as positive (correct), negative (incorrect), or neutral (not determinative of the final answer's correctness). The resulting dataset, PRM800K, contains approximately 800,000 step-level annotations across 75,000 solutions. The quality of human annotation depends heavily on annotator expertise. Mathematical step correctness often requires domain knowledge that cannot be crowd-sourced to general-purpose annotators.

Monte Carlo estimation. For each prefix of a reasoning chain (steps 1 through ), generate multiple continuations using the base model and check whether any continuation reaches the correct final answer. The proportion of continuations that succeed is used as the label for step . A step receives a high score if the model can frequently recover from that prefix and reach the correct answer; it receives a low score if the model rarely succeeds from that point onward.

More precisely, given a problem and a partial solution , we generate completions for and compute:

where is the final answer reached by the -th completion and is the correct answer. This estimated success rate becomes the training label for step . The intuition is that a step is "good" if the model can build on it to reach the right answer, and "bad" if the model consistently fails from that prefix.

The Monte Carlo approach allows automated label generation without human annotation, at the cost of computational overhead (many continuations must be generated per prefix) and potential noise (the success rate of a prefix is not the same as the correctness of the most recent step, but it correlates strongly with it). A subtle issue is that Monte Carlo labels measure the capability of the current base model to recover from a prefix, not the intrinsic correctness of the steps in that prefix. If the base model is unusually good at recovering from certain types of errors, those errors receive artificially high scores.

Given step-level labels, the PRM is trained with a classification or regression loss. For binary step correctness labels, a cross-entropy loss is standard:

where:

• is the number of steps in the reasoning chain
• is the binary label for step (1 = correct, 0 = incorrect)
• is the PRM's predicted probability that step is correct

The loss sums over all steps in the chain. Each step contributes independently to the gradient, which means the PRM learns to evaluate each step based on its local context without being dominated by the outcome at the end.

For continuous labels (such as Monte Carlo success rates), a mean squared error loss or binary cross-entropy with soft targets is used instead. In the soft-target case, the label is replaced by the Monte Carlo estimate , and the same cross-entropy formula applies. This treats the Monte Carlo estimate as a probability rather than a hard binary label, which is appropriate given the stochastic nature of the estimation procedure.

One important consideration is class weighting. In most reasoning datasets, correct steps are far more common than incorrect steps. A naive training procedure that minimizes unweighted cross-entropy on a 90-10 correct-incorrect split will converge to a model that predicts "correct" for every step, achieving 90% accuracy while being entirely useless as an evaluator. Proper class weighting or focal loss formulations are necessary to ensure the model learns to identify incorrect steps.

The distinction between outcome supervision and process supervision in training can be formalized. Let be a solution with steps, and let be the final answer.

With outcome supervision, the reward is:

where is the correct answer. The entire solution receives a single binary reward.

With process supervision, the reward at step is:

where is the problem and is the process reward model's score for step given the problem and all preceding steps.

The key difference is that process supervision provides a vector of rewards rather than a single scalar. This vector can be used in multiple ways: to directly train a policy via reinforcement learning, to rank candidate solutions step by step, or to select the best reasoning paths in a search algorithm.

When used to train a policy via reinforcement learning, process supervision typically applies a per-step policy gradient update. The policy gradient theorem gives:

where is the policy (the language model being trained), is the -th reasoning step, and is the process reward at step . Compared to the outcome reward case where would be constant across all steps and equal to the final reward, the process reward provides a differentiated signal: steps with high process reward receive positive gradient updates, while steps with low process reward receive negative ones.

One important consideration in PRM training is the relationship between the PRM and the policy model whose outputs it evaluates. In most setups, these are separate models: the policy generates candidate solutions, and the PRM evaluates their steps. But the quality of the PRM depends critically on what distribution of solutions it was trained on.

A PRM trained on solutions generated by a weak base model will see primarily simple errors and simple recoveries. When applied to solutions generated by a stronger model, it encounters different error patterns and may not generalize well. Conversely, a PRM trained on solutions from a very strong model may see too few errors to learn what incorrect steps look like.

This suggests that PRM training should be iterative: train a PRM on solutions from the current policy, use the PRM to improve the policy, then retrain the PRM on solutions from the improved policy, and so on. This bootstrap approach mirrors the iterative nature of RLHF fine-tuning more generally. In practice, maintaining this cycle is computationally expensive, so most published work uses a fixed PRM trained on a fixed dataset.

The strongest empirical results for process reward models have come from mathematical reasoning tasks. Mathematics provides several properties that make it an ideal testbed for developing and evaluating PRMs.

First, mathematical reasoning is inherently multi-step. Solving a competition problem requires applying a sequence of definitions, theorems, algebraic manipulations, and logical inferences. Each step can be independently judged for validity. This structure maps naturally onto the PRM evaluation paradigm.

Second, correctness is verifiable. Mathematical solutions either reach the correct numerical answer or they do not. This makes it possible to generate large amounts of training data by having a model produce candidate solutions and checking the final answer automatically. The MATH benchmark, AMC/AIME competitions, and GSM8K problems all provide ground-truth answers that enable automated verification.

Third, errors are localized. A mathematical error typically occurs at a specific step and produces a specific incorrect claim. This makes step-level labeling tractable: an annotator can identify the first step where the reasoning goes wrong, rather than needing to assess overall quality. In contrast, assessing the correctness of an argumentative essay at the step level would require much more subjective judgment.

Fourth, the error taxonomy is relatively well-defined. Mathematical errors fall into recognizable categories: arithmetic mistakes, incorrect theorem application, invalid algebraic manipulation, missing case analysis, and so on. Annotators with mathematical training can reliably classify steps into these categories, producing consistent labels.

These properties make mathematics the natural first domain for PRM development, and most foundational PRM research has used mathematical benchmarks. However, the principles generalize, and researchers have begun exploring PRMs for code generation, formal verification, and scientific reasoning.

One of the most direct applications of a PRM is in best-of-N search. Instead of training a new policy, you use the PRM to select the best solution from a set of candidate solutions generated by a base model.

With outcome reward, best-of-N selects the solution that the ORM scores highest based on the complete solution. This is equivalent to having the ORM rank all solutions and return the top one. With process reward, you have two natural strategies:

Step-level reranking. Score each candidate solution by taking the minimum step score across all steps. A solution with one very bad step receives a low overall score, even if most steps are excellent. This is conservative but effective at eliminating solutions with clear errors.

Step-level aggregation. Score each solution by taking the product or geometric mean of step scores. This aggregates step quality across the entire solution.

Both strategies outperform selecting by final-step score alone, because they account for the quality of the entire reasoning path, rather than only the conclusion. The minimum-step strategy has shown stronger empirical performance on mathematical benchmarks, likely because a single incorrect step in mathematics typically invalidates the entire solution regardless of how well the other steps are executed.

The relationship between best-of-N performance and the size of reveals important properties of both the PRM and the base model. Plotting accuracy against (on a log scale) produces a curve that rises steeply at first and then levels off. The height at which the curve levels off reflects the ceiling imposed by the base model's capability: no amount of selection can yield a correct solution if none of the candidates contains one. The steepness of the initial rise reflects the quality of the PRM as a selector: a better PRM selects the correct solution from fewer candidates.

A more powerful application is using the PRM to guide beam search during generation. Instead of generating complete solutions and then reranking them, you use the PRM to prune bad steps during generation. This changes the character of the search from selection among complete solutions to active guidance of the generation process.

The procedure works as follows:

1. Begin with the problem as the initial context.
2. Generate candidate next steps from the current prefix using the policy model.
3. Score each candidate step using the PRM, conditioning on the problem and all preceding steps.
4. Keep the top candidates (beam width ) and discard the rest.
5. For each surviving beam, generate more candidate next steps and repeat.
6. Continue until all beams have reached a terminal step (end of solution).
7. Select the highest-scoring complete solution from among the surviving beams.

This is similar to how beam search works in sequence generation, but the scoring function is the PRM rather than the language model's own log probability. The PRM acts as an evaluator, the language model acts as a proposer, and together they search the space of reasoning chains more effectively than either could alone.

The advantage of beam search over best-of-N is computational efficiency and search quality. Beam search explores promising paths more deeply by catching bad steps early and not wasting computation on paths that have already gone wrong. If step 3 is clearly incorrect, beam search discards that path before generating steps 4 through 20, while best-of-N would generate all 20 steps and only then reject the solution. This early pruning makes beam search substantially more efficient for fixed compute budgets.

The tradeoff is that beam search can commit to a flawed path if the PRM makes an early incorrect judgment. If the PRM assigns a high score to a subtly incorrect step 3, beam search will build on that step and potentially waste many subsequent computation steps before the error manifests as a wrong answer. Best-of-N, by contrast, assesses complete solutions and is less susceptible to this early-commitment failure mode.

The optimal beam width depends on the reliability of the PRM. With a highly accurate PRM, small (even 2 or 3) may suffice. With a noisy PRM, larger provides a safety buffer but increases computational cost. In practice, is typically set to between 4 and 16 for mathematical reasoning tasks.

At the frontier of reasoning research, process reward models are being combined with Monte Carlo Tree Search (MCTS) to create even more powerful search procedures. MCTS was developed in the context of game-playing AI (most famously in AlphaGo and AlphaZero) and provides a principled framework for searching tree-structured spaces with uncertain value estimates.

In the reasoning setting:

• Each node in the search tree represents a partial reasoning chain (a problem plus some steps).
• Expanding a node means generating candidate next steps from that prefix.
• The PRM evaluates the quality of each newly added step, giving an immediate local signal.
• Monte Carlo rollouts from a node estimate the probability of reaching the correct answer from that point, giving a longer-range value estimate.
• The MCTS algorithm balances exploration (trying new paths) with exploitation (extending high-value paths already discovered) using the UCB (Upper Confidence Bound) selection criterion.

The PRM is a value function in the MCTS framework, estimating the "value" of a partial solution (how likely it is to lead to a correct answer). This combination addresses a key weakness of both pure PRM scoring and pure Monte Carlo estimation: PRM scoring without rollouts can miss long-range dependencies, while pure Monte Carlo estimation is computationally expensive because it requires running many full completions. MCTS with PRM-guided rollouts finds a middle ground, using the PRM to quickly assess local step quality while using sparse Monte Carlo rollouts to calibrate longer-range value estimates.

This combination has shown impressive results in recent work. Models that reason via MCTS with PRM guidance substantially outperform both greedy decoding and standard beam search at comparable compute budgets, especially on the hardest mathematical problems where the search space is largest and step-level guidance is most valuable.

Let us trace through a concrete example to see how a PRM evaluates a multi-step solution. This example illustrates both the information a PRM provides and how it differs from outcome reward.

Consider the problem: Solve for : .

Here is a candidate solution with five steps:

Step 1: Divide both sides by 2 to simplify: .

Step 2: Factor the quadratic: .

Step 3: Apply the zero product property: or .

Step 4: Solve each equation: or .

Step 5: Check: . And .

A well-trained PRM would assign high scores (say, 0.95 or higher) to all five steps. The derivation is mathematically correct at every stage. Now consider an alternative solution with an error:

Step 1: Divide both sides by 2: . (Score: 0.94, correct)

Step 2: Factor incorrectly: . Error: . (Score: 0.08, incorrect)

Step 3: Apply the zero product property: . (Score: 0.71, conditionally correct given step 2)

Step 4: Solve: . (Score: 0.72, conditionally correct given step 3)

Step 5: Check: . Inconsistency detected. (Score: 0.15, reveals error)

The PRM assigns a low score to Step 2 because the factorization does not equal . An ORM would also correctly reject this solution (the final answer is wrong), but it provides no information about where the error occurred. The PRM's signal is more informative: Step 2 is where the reasoning breaks down.

Notice also that Steps 3 and 4 receive relatively high scores despite being based on the incorrect Step 2. This is appropriate: given the (incorrect) factorization from Step 2, applying the zero product property and solving are valid operations. The PRM correctly attributes the error to Step 2 rather than penalizing Steps 3 and 4 for following logically from a flawed premise.

Now consider a third, more insidious scenario: a solution that reaches the correct numerical answer through an algebraically incorrect path. Suppose a student incorrectly applies the quadratic formula with a sign error but then makes a compensating arithmetic mistake, arriving at and despite the flawed derivation. An ORM would reward this solution identically to the correct solution, because the final answers match. A PRM would detect the error at the step where the sign mistake occurs and correctly identify this as a flawed derivation despite the correct answer. Training against an ORM that rewards such solutions teaches the model to produce compensating errors; training against a PRM discourages this pattern entirely.

To make the Monte Carlo label generation procedure concrete, consider how we would estimate the correctness probability for Step 2 in the above example. Starting from the prefix (problem + Step 1 + the incorrect Step 2), we generate completions of steps 3 through 5 using the base policy model.

From a prefix that includes the incorrect factorization , almost all completions will arrive at via valid subsequent steps. The final answer will consistently be , which is incorrect. The Monte Carlo estimate becomes:

The estimate of approximately 0.02 correctly identifies Step 2 as a problematic step. The model can almost never recover from an incorrect factorization and arrive at the correct final answer.

Compare this to the Monte Carlo estimate for Step 1. Starting from the correct prefix (problem + Step 1), the base model can frequently complete the solution correctly:

A strong base model succeeds 80% of the time from this point. The Monte Carlo label for Step 1 is 0.80, correctly indicating that this prefix is in good shape. The gap between 0.80 and 0.02 across these two steps provides a training signal that the PRM can learn from: steps that preserve the ability to complete correctly receive high labels, steps that destroy that ability receive low labels.

In this section, we implement a simplified process reward model. We use a small feedforward network as a proxy for the transformer backbone that would power a full-scale PRM, to illustrate the key components of the training pipeline.

We begin by generating synthetic step-level training data. In practice, you would use human annotations or Monte Carlo rollouts, but for this walkthrough we create a toy dataset to illustrate the mechanics.

Building the PRM DatasetLink Copied

The PRM takes a concatenated context-plus-step vector and produces a single probability that the step is correct. In a full-scale implementation this would be a transformer computing attention over token sequences; here we use a feedforward network to demonstrate the training mechanics clearly.

The architecture uses Layer Normalization after the first linear layer, which stabilizes training by normalizing the hidden representations before the nonlinearity. The GELU activation function is used throughout, consistent with modern language model practice. The final sigmoid ensures the output is a probability in , suitable for binary cross-entropy training.

In a production PRM, this feedforward network would be replaced by a transformer encoder that attends to the full sequence of problem tokens and step tokens, building contextual representations that capture the semantic content of each step in relation to the full reasoning chain. The feedforward network here is a stand-in that demonstrates the training mechanics without requiring the computational resources of a full transformer training run.

Training LoopLink Copied

We now show how to use the trained PRM to score candidate solutions using two aggregation strategies.

The key design choices for a process reward model are:

• Aggregation strategy: Min-step is conservative and typically performs better for math; geometric mean is more forgiving and may suit exploratory reasoning
• Label generation method: Human annotation produces high-quality labels; Monte Carlo estimation scales automatically but introduces noise
• Step granularity: Coarser steps are easier to annotate; finer steps provide more precise credit assignment
• Loss function: Binary cross-entropy for human labels; soft targets or MSE for Monte Carlo success rates. Class weighting is important when incorrect steps are rare.
• Backbone model: A larger, more capable backbone generalizes better across problem types but increases the computational cost of both training and inference.

VisualizationsLink Copied

Understanding process reward models in isolation gives you the mechanics, but their full significance becomes clear when you see how they fit into broader approaches to training capable, reliable AI systems.

Standard RLHF trains a reward model to capture human preference over complete outputs and then fine-tunes the policy via PPO or a similar algorithm to maximize that reward. The PRM is a direct extension of this paradigm to the step level. Instead of one reward model that assesses complete outputs, you have a reward model that assesses partial outputs at each step boundary.

This extension changes the character of the RL training loop. With standard RLHF, the reward signal is sparse: the policy generates a full response and receives one reward. With PRM-guided RL, the policy generates steps one at a time and receives a reward at each step. This denser feedback loop can accelerate learning substantially, especially for tasks where the policy makes many small decisions that contribute to the final quality.

In practice, PRM-guided RL has been implemented using variants of PPO where the per-step reward is the PRM score rather than a terminal reward. The KL penalty between the policy and the reference model is typically still applied at the sequence level to prevent the policy from diverging too far from the pretrained model's behavior. This creates a training objective that balances step-level correctness (rewarded by the PRM) with overall fluency and coherence (preserved by the KL penalty).

Scalable oversight is the challenge of assessing and improving AI systems on tasks where human evaluation is expensive, time-consuming, or requires expertise that is not widely available. As AI systems become capable of producing long, technically complex reasoning, the bottleneck shifts from model capability to evaluation quality.

Process reward models contribute to scalable oversight by providing automated step-level evaluation. Rather than requiring a human expert to evaluate every solution to a competition mathematics problem, a well-trained PRM can identify the step where reasoning goes wrong, flagging only those solutions for human review. This allows human annotators to focus their effort where it is most needed.

However, PRMs are not a complete solution to scalable oversight. A PRM trained on human-labeled data inherits the biases and knowledge limitations of those human annotators. For problems that exceed human expertise, the PRM cannot provide reliable evaluation. This is a fundamental challenge: the hardest problems, where AI assistance would be most valuable, are also the problems where step-level human annotation is most difficult and expensive.

The combination of PRMs with automated verification, where the correctness of a step can be checked symbolically or computationally, partially addresses this challenge. If steps in a formal proof can be checked by a theorem prover, or if code steps can be tested by running them, the PRM can be augmented or replaced by a verifier that does not require human expertise. This integration of learned reward models with formal verification tools is an active area of research.

Constitutional AI, developed at Anthropic, uses a set of principles to guide model behavior through self-critique and revision. The model generates responses, critiques them according to the principles, and revises based on the critique. This process bears structural resemblance to PRM-guided training: in both cases, a critic assesses intermediate states and the generator uses that feedback to improve.

The key difference is the target: constitutional AI focuses on alignment properties (helpfulness, harmlessness, honesty), while PRMs focus on logical correctness of reasoning steps. But the underlying insight is the same: decomposing evaluation into structured, intermediate signals enables more precise and effective training than whole-response evaluation alone.

Process reward models represent a significant advance in training signal quality for reasoning tasks, but they come with substantial limitations that constrain their practical applicability.

The most fundamental limitation is the cost of step-level annotations. Human annotation of individual reasoning steps is significantly more expensive than annotating final answers. A human rater must read and understand each step, verify its logical validity given the preceding context, and make a correctness judgment. For complex mathematical proofs or multi-step code generation, this is cognitively demanding work that cannot be delegated to non-experts.

The OpenAI PRM800K dataset, one of the most widely used PRM training resources, required approximately 800,000 step-level annotations from human mathematicians to cover the MATH benchmark. Scaling this to other domains, longer reasoning chains, or more complex problems requires proportional increases in annotation cost. The economics of annotation become a fundamental bottleneck: you need expert annotations at scale, but expert time is both expensive and limited in supply.

The Monte Carlo estimation approach alleviates this somewhat by automating label generation, but introduces a different cost: computational overhead. Generating hundreds of continuations per reasoning prefix requires significant inference compute. The quality of Monte Carlo labels also depends on the quality of the base model: if the base model is weak, Monte Carlo rollouts are dominated by noise and produce unreliable labels. Worse, if the base model has systematic biases (always preferring a certain incorrect approach), the Monte Carlo estimates will reflect those biases rather than true step correctness. The Monte Carlo approach effectively bootstraps the quality of the PRM from the quality of the base model, which creates a chicken-and-egg problem early in development.

A PRM trained on one domain or problem type may not generalize well to others. A PRM trained on competition mathematics does not automatically become a useful evaluator for code generation, logical reasoning, or scientific problem solving. The step correctness criteria differ significantly across domains, and the surface patterns that correlate with correctness in one domain may not transfer to another.

This is a practical limitation because building a PRM requires both a large base policy (to generate training data) and a large annotated dataset (to train the PRM). Both requirements are domain-specific. As a result, PRMs have primarily been demonstrated in narrow, well-defined domains like mathematics and code generation. Extending to broader reasoning domains, or building a general-purpose PRM that works across many different types of problems, remains an open challenge.

Even within mathematics, generalization across subfields is non-trivial. A PRM trained predominantly on algebra may not perform well on combinatorics, number theory, or calculus, even though all are mathematical reasoning tasks. The notation conventions, typical error patterns, and valid proof strategies differ enough across subfields that domain-specific training may be necessary.

A PRM requires that the reasoning chain be decomposed into discrete steps that can be individually evaluated. This decomposition is not always natural. Mathematical proofs have clear step structure; but other forms of reasoning (creative writing planning, scientific hypothesis formation, open-ended question answering) are harder to decompose into steps with independent correctness criteria.

Even within mathematics, the granularity of steps affects PRM quality. A coarse decomposition (few, long steps) may bundle multiple reasoning operations into a single step, making it difficult to localize errors. A fine decomposition (many, short steps) increases annotation cost and may produce trivially correct micro-steps that provide little training signal.

A recurring challenge in reward model-based training is overoptimization: the policy learns to maximize the reward model's score without improving the underlying capability. This problem appears in standard RLHF (where it was studied in detail by Gao et al., 2023) and is equally present in PRM-guided training.

A policy trained against a PRM can learn to produce steps that the PRM predicts are correct without those steps being valid. If the PRM has systematic biases (for example, rewarding verbose steps, steps with a lot of mathematical notation, or steps that match surface patterns from the training distribution), the policy will exploit those biases. Over many training iterations, the policy and PRM diverge from the underlying ground truth, and performance can degrade despite high PRM scores.

This phenomenon is sometimes called Goodhart's Law in the ML context: when a measure becomes a target, it ceases to be a good measure. The PRM's score was correlated with correctness when it was trained, but once the policy actively optimizes against that score, the correlation degrades. The policy finds adversarial examples for the PRM, generating solutions that receive high step scores but are factually incorrect.

One mitigation is to periodically update the PRM using new data generated by the current policy. By continuously retraining the PRM on the distribution the policy produces, you reduce the gap between what the PRM was trained on and what the policy generates. Another mitigation is to combine PRM guidance with verifiable correctness checks where possible (for example, running code to verify that a generated program produces the correct output). Neither approach fully eliminates overoptimization, and this remains an active area of research.

A PRM assigns a low score to any incorrect step, even if a skilled solver could recover from that step and reach the correct answer. In some reasoning contexts, detours through incorrect intermediate claims are a natural part of exploration. A student solving a novel problem might try an approach that does not work out, recognize the failure, and backtrack. A PRM that penalizes these exploratory steps may discourage the kind of flexible reasoning that leads to creative solutions.

This tension between strict step correctness and exploratory reasoning is not fully resolved in current PRM research. Most PRMs are trained to evaluate steps in a forward-only manner, given a specific chain, rather than to evaluate the overall quality of a reasoning strategy.

Beyond the general limitations, PRMs face challenges specific to their primary application domain:

Symbolic ambiguity. Mathematical notation admits many equivalent forms. The PRM must recognize that different surface forms of the same expression are all valid.

Proof branching. Many mathematical results can be proven by multiple valid approaches. A PRM trained predominantly on one standard proof technique may incorrectly penalize steps from an alternative but equally valid proof strategy.

Implicit assumptions. Mathematical steps often rely on implicit assumptions (for example, that a variable is positive or that a function is continuous) that are not stated explicitly. Evaluating whether a step is valid requires understanding these implicit conditions from context, which is challenging for current language model-based PRMs. A step that applies a theorem whose conditions have not been verified should receive a low score, but detecting this requires the PRM to track which conditions have and have not been established throughout the chain.

Notation-sensitive errors. Some mathematical errors are only visible when you expand implicit operations. A PRM that processes mathematical text at the surface level may not detect an error that is only apparent when you work through the detailed arithmetic or algebraic manipulation behind a compact expression.

Beyond the conceptual limitations, PRMs face practical challenges when deployed in real systems.

Inference cost is significant. Each step evaluation requires a forward pass through the PRM backbone, which is typically a large language model. In beam search with a beam width of 8 and a 20-step reasoning chain, you run the PRM 160 times per problem. At scale, this inference cost must be weighed against the accuracy improvement it provides. For many applications, the compute budget required for PRM-guided search exceeds what is available at inference time.

Latency is a related concern. PRM-guided beam search has higher latency than greedy decoding because steps must be evaluated before the next step is generated. For interactive applications where users expect low-latency responses, this creates a tension between reasoning quality and response speed. Distilling the PRM's quality improvements into the policy model, training the policy to reason like PRM-guided search would but without running the PRM at inference time, is one approach to resolving this tension, and it is an active research direction.

Step boundary detection is also non-trivial in practice. The PRM must know where one step ends and the next begins. For well-formatted mathematical solutions, step boundaries are usually clear. For more conversational reasoning, the boundaries are ambiguous. Training a secondary model to detect step boundaries, or requiring the policy to format its reasoning in a structured way that makes boundaries explicit, adds complexity to the overall system.

Process reward models provide step-level evaluation of reasoning chains, addressing the credit assignment limitations of outcome-only reward signals. They assign a correctness score to each step in a multi-step solution, rather than evaluating only the final answer. This dense reward signal enables more precise training feedback and more effective search procedures.

The core distinction between outcome reward and process reward lies in granularity. Outcome reward treats the reasoning chain as a black box and assigns a single signal based on the final answer. Process reward evaluates each step independently, enabling the model to learn from errors at the point where they occur rather than at the end of a long sequence. This distinction matters most for long-horizon reasoning tasks, where sparse rewards are difficult to optimize and where coincidental correctness and reward hacking are significant failure modes.

Step-level labels for PRM training are generated either through human annotation (high quality, expensive, domain-expert required) or Monte Carlo estimation (automated, computationally intensive, quality tied to base model capability). Both approaches have been shown to produce useful PRMs, with human annotation yielding higher quality labels at substantially greater cost. The PRM800K dataset from Lightman et al. established human annotation at scale as a viable path for mathematical reasoning.

PRMs are applied in three main search settings: best-of-N reranking (generate complete solutions and select the highest-scoring one), beam search guidance (prune candidate steps during generation to focus compute on promising paths), and Monte Carlo Tree Search (combine step-level evaluation with long-range rollouts for principled exploration of the reasoning space). The minimum-step aggregation strategy, which rates a solution by its weakest step, has shown stronger empirical performance than geometric mean aggregation for mathematical tasks where a single incorrect step invalidates the entire solution.

The relationship between PRMs and broader training paradigms illuminates their role in the AI development field. They extend RLHF to the step level, enabling denser feedback loops for RL training. They contribute to scalable oversight by allowing automated flagging of problematic reasoning steps for human review. And they embody the same structural insight as constitutional AI and debate-based training: decomposing evaluation into structured, intermediate signals produces better training outcomes than whole-response assessment alone.

The key limitations include the high cost of step-level annotation, poor out-of-distribution generalization, sensitivity to step decomposition granularity, susceptibility to reward model overoptimization, and the tension between penalizing incorrect steps and allowing exploratory reasoning strategies. Practical deployment challenges include inference latency, computational cost, and step boundary detection. These limitations define active areas of research: how to generate step-level supervision at scale without expensive human annotation, how to generalize across reasoning domains, how to prevent overoptimization as policies improve, and how to handle the subtleties of exploratory and non-standard reasoning.

Process reward models represent an important step toward training systems that produce correct answers and reason correctly along the way. The broader principle, that structured intermediate feedback outperforms sparse terminal feedback for complex sequential tasks, extends well beyond mathematical reasoning. We will revisit reward model design and structured evaluation in upcoming chapters on scalable oversight and constitutional AI, where similar themes of process versus outcome evaluation appear in different forms.

QuizLink Copied