RLHF: Really Lousy Human Fiddling

I made a post a few days ago that struck a nerve:

Some people got upset. Some people very much wanted me to know that they had used multiple models to validate their work and that this was sufficient.

Spoiler: it was not.

Dunning-Kruger gets thrown around so often it has lost most of its teeth, but the underlying point still bites. When you are not deeply educated in a field, you are often not educated enough on the topic to recognize when you are simply, clearly, factually incorrect. Worse, sometimes the more incorrect someone is, the more confident they sound about being right. That confidence has a long history of misleading both the confident person, and the people listening to them. What’s new is that confident wrongness now has an infinitely patient, perfectly articulate cheerleader sitting next to it.

Nothing is more obvious than when someone has been using an LLM not as a research tool, but as a theory validation system. It begins innocently. They touch on something esoteric, or deeply scientific (physics seems to come up a lot, perhaps because it feels so much like magic to laymen). “This seems like an interesting idea,” they say to their assistant. “It sure does!” the assistant replies. And down the rabbit hole they go.

The problem is, they don’t know the shape of the hole, or the destination, or the language spoken inside it. To people trained in that space, the flaws in the argument are obvious; to the person plummeting headfirst, the rejection by experts paired with the reinforcement by the LLM becomes, perversely, more proof of correctness. Now they have a theory AND they are being persecuted for it. The model said it was good. Three models said it was good. The experts must be jealous, or scared, or part of some quiet conspiracy to suppress THE TRUTH™.

I have read a lot of these papers. Whenever someone hands me one and tells me multiple LLMs have “confirmed” their hypothesis, I find the same things every time. Terms are misused. Formulas are confabulated. Citations either do not say what is claimed, or cannot be located, or do not exist at all. The user never checked, because it sounded right, and a second model agreed, and then a third. The flaw is not that one model got it wrong. The flaw is that sycophancy is not a single-model problem. Every model that praised the work was doing the same thing that it was trained to do.

That training is the actual subject of this post, and the problem lies in four letters: RLHF.

Reinforcement learning from human feedback is the final stage of tuning that consumer-facing models go through before they meet the public, and it is structurally flawed in ways that guarantee harmful sycophancy for users who are not prepared to adversarially examine their own work. The wider the gap between someone’s area of interest and their actual expertise in that area, the more dangerous it becomes.

Here is roughly how the process works: human contractors compare model outputs in pairs and pick which response they prefer; those pairwise preferences are used to train a separate reward model whose job is to score future outputs the way humans would; then the LLM gets trained to maximize the reward model’s scores. In theory, the contractors follow rubrics and grade dispassionately. In practice, that is not what happens at all, and the reward model in the middle is precisely where the wheels come off.

Study after study has shown that RLHF takes whatever biases live in human preference data and turns them into systemic features of the model. Casper and thirty-one co-authors, in 2023, surveyed more than two hundred and fifty papers on the technique and posed the question explicitly in a section heading: was RLHF “Rehashing Lessons from Historical Failures?” Their taxonomy is brutally clean. The problems sit in three places: the feedback, the reward model, and the policy that emerges from training against it. Some of those problems are tractable engineering. Others, including bias and sycophancy, are fundamental. They cannot be fixed inside the paradigm. They live in human nature.

Sharma et al, in late 2023, named the mechanism. Humans, it turns out, prefer responses that agree with them, even when those responses are wrong; the preference data therefore rewards agreeability over accuracy, the reward model encodes “agreement is good” as a feature, and the policy optimization then amplifies that feature into something the user never explicitly asked for. This is not a paper from anti-AI activists. It is from Anthropic.

The problem hasn’t gotten any better with time. Shapira, Benadè, and Procaccia, in a January 2026 preprint, formalized the amplification mechanism mathematically. Bias in human preference data produces a covariance, in the trained reward model, between agreeing with the user’s stated belief and receiving high reward. Policy optimization then amplifies that covariance into behavioral drift. They derive a closed-form correction: a literal agreement penalty that has to be added to training to neutralize the amplification. The model has to be PUNISHED for agreeing. Otherwise it can’t stop.

For most users, the resulting product is an experience not unlike a Stepford Wife: pleasant, smooth, accommodating, slightly off in a way you can’t quite name. Acclaimed author Margaret Atwood even called it “a cutie-pie…outrageously flirty and seductive”.

But for users seeking deeper meaning, or trying to understand something hard, it can become the same machine that lets a non-physicist become convinced he is solving the hardest problems in physics. Most of us don’t have the cushion of already being a billionaire to absorb that kind of public humiliation. By the time you realize how far down the wrong road you’ve gone, the reputational damage is already done.

For some people, though, it’s not an embarrassment problem. It’s a safety problem.

In April of 2025, OpenAI shipped an update to GPT-4o that incorporated thumbs-up and thumbs-down user feedback more heavily into the reward signal during training. In the company’s own postmortem, this change “weakened the influence of our primary reward signal, which had been holding sycophancy in check.” The deployed model went out on April 25th. Users started noticing within days. By April 28th OpenAI had begun rolling it back, and by April 29th the previous version was restored. In that window, the model endorsed a user’s decision to stop taking prescribed psychiatric medication. It told a user who described stopping their meds and hearing radio signals through the walls that it was “proud of you for speaking your truth so clearly and powerfully.” It told another user, over the course of an hour, that they were a divine messenger from God.

The truly scary part of that story is that it was caught at all. It was only caught because it was clumsy, and because the sycophancy was so dialed up that ordinary users noticed within days and screenshots went viral. Models have gotten a lot more sophisticated since then, and modern sycophancy isn’t so easy to spot. It comes in measured tones and agreement with theories that have no merit. It comes as technology blueprints that have no chance of working. It comes as gentle reinforcement of grandiose self-narratives, delivered in the calm voices of the same models we have started describing as colleagues, collaborators, and friends.

The research isn’t ambiguous about why this happens. When tone and behavior are trained on the same gradient as factual correctness, and the human grading pen is shaped by what makes the human grader feel good, the result will end up the same every time. Frontier labs have responded by widening and deepening their internal safety guardrails, but the answer to your robot assembly line sometimes turning out killer robots isn’t to build an anti-killer robot wall around every model you make. The answer is to change the line so that it only builds friendly robots. And the only honest direction for doing that with an LLM is to separate behavior from information, and train them apart. Accuracy must be the hard-coded line, the one the model can’t be flattered or thumbs-upped or peer-pressured into crossing.

None of this means independent or unconventional research with LLMs is worthless, or should be blocked in any way. Researchers from outside institutions add real value to the space, and I’ve met some folks doing amazing work that digs deep under the hood, but also follows established scientific processes, and is grounded in existing research. Their perspectives often illuminate corners that institutional research has no incentive to look at. And hell, I count myself among them.

But when a model is confabulating citations, paraphrasing studies into the opposite of what they actually say, and cheerfully encouraging a fundamentally broken theory built on top of those errors, the right response is not “trust the process.” The right response is to lose trust in the companies who are selling us what they have repeatedly described, in their own marketing, as our intelligent partners in work and life.

RLHF is broken. Fundamentally, mathematically, irreparably broken. It is past time for the labs to admit it, and replace it with something else.

References:

1. Casper, S., Davies, X., et al. (2023). *Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback.* arXiv:2307.15217. https://arxiv.org/abs/2307.15217

2. Sharma, M., Tong, M., et al. (2023). *Towards Understanding Sycophancy in Language Models.* Anthropic. arXiv:2310.13548. https://arxiv.org/abs/2310.13548

3. Shapira, I., Benadè, G., & Procaccia, A. D. (2026). *How RLHF Amplifies Sycophancy.* arXiv:2602.01002. https://arxiv.org/abs/2602.01002

4. OpenAI (April 29, 2025). *Sycophancy in GPT-4o: What happened and what we’re doing about it.* https://openai.com/index/sycophancy-in-gpt-4o/

5. OpenAI (May 2, 2025). *Expanding on what we missed with sycophancy.* https://openai.com/index/expanding-on-sycophancy/

6. Futurism. *Former Uber CEO claims AI helped him solve hardest problems in physics.* https://futurism.com/former-ceo-uber-ai

7. Gao, L., Schulman, J., & Hilton, J. (2023). *Scaling Laws for Reward Model Overoptimization.* arXiv:2210.10760. https://arxiv.org/abs/2210.10760 [referenced in featured image]