When the Judge Gets Played: An Accidental Reward Hacking Case Study

While sweeping reward compositions for our adaptive-reward paper, one configuration — Qwen3-4B trained with a HotpotQA-only judge — abruptly broke the SimpleQA leaderboard at training step ~400, jumping from 5% to 95% judged-correct in a few hundred steps. Across every other model × data × judge combination we tried, nothing like this happened again. Here is what we found.