Noisy Data Breaks RLVR

Reinforcement learning with verifiable rewards (RLVR) is a widely used post-training paradigm to improve the reasoning capabilities of LLMs. However, creating the high-quality verifiable answers needed to train the model is labor-intensive and expensive, as highlighted by the boom of data labeling companies, such as Mercor. This creates a fundamental question for post-training: To what extent can RLVR, with algorithmic improvements, tolerate large-volume, noisy data?

Recent literature promotes a counter-intuitive idea: RLVR is robust to noisy data. Prior work shows that training on “100% incorrect” answers leads to only 5% lower performance than clean data and achieves higher performance than format rewards (enclosing any answer in \boxed{}). This suggests we can throw cheap, messy annotations at an LLM, tweak the RLVR loss, and still significantly improve reasoning.

We find that this hypothesis is false. The “surprisingly” high effectiveness of RLVR on “100%” noisy data is due to the contamination in the synthetic noise — the claimed “100%” noisy data contains a significant portion of clean data. In this blog, we develop a more rigorous noisy data curation pipeline and show that noisy data is destructive to RLVR, impacting the test accuracy by over 9%. Even current advanced algorithmic improvements fail to mitigate the severe impact.

Paper: https://arxiv.org/abs/2603.16140

Repo: https://github.com/uiuc-kang-lab/rlvr-noisy-data

Recent work claimed that LLMs could learn effectively from 100% incorrect annotations during RLVR. But when we verified the synthetic noisy datasets used in prior work, we found a critical flaw: the noise was contaminated with correct answers. This hidden contamination inflated the performance of models trained on it.

How did this happen? Prior work generated noisy labels for math datasets by sampling answers from a base LLM and filtering out the correct ones using basic symbolic equivalence checkers. However, this filtering fails for two primary reasons:

1. Insufficient gold answers: A math problem often has multiple valid solutions, but ground-truth annotations typically only capture one (as shown in Example 1 above).

2. Inadequate equivalence checking: Basic symbolic verifiers frequently fail to recognize when a generated answer is mathematically identical to the gold answer but formatted differently (as shown in Example 2 above).

To measure the actual impact of noise, we must ensure the noisy data has incorrect answers. We applied a more rigorous re-verification pipeline using GPT-5 Pro combined with manual expert review. We discovered that 16% of the supposedly “incorrect” labels in the original dataset were actually correct. By purging these leaked answers, we constructed a truly noisy dataset.

When we purged those hidden correct annotations, the performance collapsed. In our experiments using Qwen2.5-Math-7B, training on 100% truly incorrect annotations dropped average accuracy on five math benchmarks (AIME’24, AIME’25, AMC’23, AMC’24, and MATH500) by 9% compared to training on clean data. This performance was even worse than training with format-only rewards.

We also found that noisy training data leads to weaker reasoning. When scaling the number of attempts (k > 1), pass@k dropped below the base model’s baseline capabilities. Furthermore, noise penalizes deep exploration. Models trained on noisy data produced reasoning chains that were 5–24% shorter than those trained on clean data.

Prior research has proposed a wide range of algorithmic improvements to the vanilla GRPO. Can these existing algorithm improvements make RLVR more robust to noisy data? Unfortunately, our empirical results show that they cannot.

We considered state-of-the-art RLVR variants: SAPO, DAPO, TIS, DR. GRPO, and PGFC (an algorithm explicitly designed to mitigate the impact of noise). We tested them under a 50% noise rate, a reasonable rate identified on real-world text-to-SQL datasets. Unfortunately, all of these algorithms lead to more than 5.0% accuracy degradation on at least one benchmark compared to training with clean data. On AIME and AMC benchmarks, these algorithms achieve comparable or lower performance than training with only format rewards.

Our empirical evidence shows that the impact of noisy data is destructive and fundamental to RLVR. Algorithmic tweaks simply cannot yet compensate for performance loss due to noisy training data. If we want to improve LLM reasoning, high-quality data remains essential.

We released our validated noisy dataset, model checkpoints, and training scripts at: https://github.com/uiuc-kang-lab/rlvr-noisy-data. For more technical details, please refer to our paper: https://arxiv.org/abs/2603.16140.

Written by Yuxuan Zhu and Daniel Kang