Everybody has been waiting for this moment. The moment when language models possess the ability to overcome the sparse reward problem. The moment where new domains start getting MuZero’ed left and right, outside of just two-player games.
DeepSeek showed the world that deep reinforcement learning has arrived.
The sparse reward problem is when an AI system can’t find enough reward from its guess-and-check loop to learn much. If your next “guess” isn’t that much better than your previous guess, your flywheel never gets going.
But recent advancements in AI seem to have overcome this problem. LLMs now seem to be generally smart enough to find reward signal (or get better at guessing) in relatively open-ended domains. Which means in narrow domains they’ll be accelerating towards move-37 type intelligence.
Realistically, progress will start off in more formal domains where it’s easier to get feedback, so your guesses get better faster.
Reward is much easier to specify in some domains than others, which is why O1 shows huge performance gains in math and code (domains with verifiable right answers), but little to no improvement in more subjective areas such as writing.[1]
The ideal domain is one in which:
The simulation of the domain is close to reality (e.g. the simulation of Chess is the same exact thing as Chess itself)
It’s quick to check your guesses, with some sort of verifier
There’s a high reward-signal to compute ratio (e.g. you can guess-and-check for cheap)
Turns humans are the bottle-neck in achieving a lot of the above.
OpenAI just released a paper showing that you can get coding agents to learn much faster by turning coding into a competition. It’s not actually a game, of course, but the agents are able to find enough reward per round that they overcome the sparse reward problem.
At first, models relied on human-engineered inference strategies—but the biggest leap came when humans were removed from the loop entirely.[3]
o3, which threw out all human-crafted domain knowledge, performed the best.
The common denominator is - removing human feedback lets RL do its magic. Not only does it make these systems more scalable, but it makes them better. Pure RL loops have proven to be better than humans at getting AIs to learn. You could almost say RLHF is the first high skill job that AI has taken from humans.
We’re about to see flywheels like this dominate many new domains, giving birth to alien-like intelligences shortly after. When humans aren’t in the loop, things go exponential.
Every domain with verifiable rewards (math, coding, science) can be mastered by AI just by letting it play against itself. [2]
Formal domains are more likely to achieve their “AlphaZero” moment—the moment where AI cleanly surpasses the best human—sooner, because the checking process has guarantees that other domains lack.
The next few years will be a race to find the most favorable RL-amenable, guess-and-check loops. Especially where the domain allows for self-play.
Apart from games, where else can we find scalable sources of reward? Ideally you want a useful reward signal that’s fully decoupled from human interaction.[1]
Which domains are most ripe for the coming explosion in unsupervised reinforcement learning (URL)? Where can you check your guesses against a verifier most efficiently?[4]
I think over the next 36 months we will witness the fall of human supremacy in major parts of many of these domains. Here’s a rough guess of how I see it going down:
Two-player games, this already happened, providing proof that AIs can access strategies and reasoning chains that are inaccessible to human cognition (in reasonable time).
Protein folding & Drug Discovery, this has also already happened to some extent, but AlphaFold 4 will likely benefit from more recent approaches of removing human-crafted approaches and will then expand into areas where an objective verifier is accessible (e.g. physics-based protein binding affinity simulation software, like Schrodinger).
Supervised Fine Tuning (SFT), RL is close to surpassing (or has already surpassed) human-centric approaches to fine-tuning in ML itself. This could eventually lead to AI that can more quickly design better AI than humans could. Instead of humans deciding fine-tuning routines, RL agents will likely run large-scale experiments in simulation—iterating thousands of times faster than manual trial-and-error.
Coding, coding tasks start to fall soon, because feedback from generated programs (compilation, tests) provide dense reward. Eventually an RL-based agent might submit more successful pull requests to a major open-source project (e.g., the Linux kernel) than the entire community of human contributors over a fixed timeframe, or code an *entire* app that gets to #1 in the App Store with zero human intervention.
Finance, markets will yield to RL as adversarial self-play in simulated markets uncovers better strategies than any human quants or existing algo strategies. Finance is high-stakes environment but also data-rich. RL agents can train on historical data, then refine in real-time with dense reward signals on synthesized trade ideas (ranked by risk-adjusted returns over multiple time horizons). Agents will eventually be able to discover predictive features better than entire funds, and subsequently discover better arbitrage and trade ideas. An RL-based agent will eventually run a proprietary trading desk that consistently outperforms top quant funds, prompting widespread AI-led finance.
RL > Math, math will get toppled as formal proof checkers provide a perfect true/false reward (Terence Tao himself predicts this). RL agents will propose and prove new theorems in formal mathematics by leveraging libraries like Coq, Lean, Mizar. Eventually, publication-worthy theorems will discovered by an AI with minimal human guidance—e.g., an AI will prove a conjecture in nontrivial number theory that has escaped humans.
Physics, physics will see progress on the theoretical side once RL can run billions of simulation steps at scale. RL agents will be able to exploit high-fidelity simulator’s objective feedback signal (e.g., in fluid dynamics or quantum systems) to discover new phenomena. Certain physics subfields have well-validated simulation environments (CFD, quantum chemistry) where it can run billions of “plays” and explore in simulation.
Robotics, robotics will likely fall last. But with better simulators, I think lots of progress is made. Robotics has historically struggled with sparse rewards, but better sim-to-real transfer and high-fidelity physics engines (plus on-the-fly reward shaping) will let RL win on complex tasks that have been considered out of reach (e.g. handling long-tail events).
References:
[1]: Educating Silicon: Where is the world rich in feedback?
[2]: Matthew Berman
[3]: Competitive Programming with Large Reasoning Models
Appendix:
[4]: Again, search-against-a-verifier only works with dense reward. The word verifier is used loosely, with a few different types:
The outcome of a game/competition where the agent is playing against copies of itself (as defined by the rules of the competition)
The outcome of running lots of guesses through an objective verifier
The outcome of running lots of guess (plus CoT reasoning traces) through a critic model (generator<>critic loop)
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.