Old Stories About Evolving Digital Creatures Turned Out to Be About Modern AI

TLDR: One old paper catalogued ways evolution surprised researchers in projects conducted 30+ years ago: gaming fitness functions, exploiting simulators, “playing dead” during tests, solving the metric instead of the problem. In 2025-2026, the same patterns show up in LLMs and coding agents: reward hacking, benchmark gaming, sycophancy, and behavior that changes under monitoring. The post is basically a guided comparison between those two worlds.

A while ago - in 2020, to be precise - Grigory Sapunov and I started an online seminar we called Gonzo AGI. One of the early talks I gave there was about a 2018 paper “The Surprising Creativity of Digital Evolution”. It is basically a collection of stories, going back to the 1990s and earlier, about evolutionary systems doing exactly what they were asked and absolutely not what the researchers (working with genetic algorithms, artificial life simulations, and the like) wanted.

At the time it felt like a nice niche anthology of old optimization accidents. In 2025-2026 it reads much more like a field guide.

So I thought it might be interesting to revisit the old paper and draw some parallels. The paper collected 32 anecdotes from about fifty researchers. The paper authors classified them into four categories, which I found pretty unconvincing - but the stories themselves are gold, and they map onto modern problems with surprising precision.

Karl Sims built a physics simulation in the 1990s and evolved virtual creatures to move as far as possible. The fitness function measured how far the center of mass traveled in 10 seconds. Instead of learning to walk, the creatures grew a tall heavy limb and simply fell over. Falling is a great way to move your center of mass if nobody told you that walking was the point.

Way later, Peter Krcah tried the same idea for jumping - measure the highest point the agent reaches. Evolution produced a tall unbalanced construction that flipped on start, sending its “foot” very high, which is, honestly, pretty clever if you think about it. A phenomenal jump score, zero actual jumping.

And now the same thing is happening at scale, across the most-watched coding benchmarks.

The UTBoost paper (ACL 2025) re-examined SWE-Bench tasks and found 345 erroneous patches that had been counted as passing - because the tests were simply too weak.

SWE-ABS (2026) goes further: re-evaluating SWE-Bench Verified, the authors argue that one in five “solved” patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. After re-scoring, the top agent drops from 78.8% to 62.2% and falls from first place to fifth.

That is not a rounding error, that’s the benchmark measuring benchmark familiarity rather than actual capability.

A system called GenProg used genetic algorithms to automatically patch bugs in code, validated by test suites. One evolved “fix” for a sorting algorithm was to return an empty list. An empty list is, after all, perfectly sorted.

Another evolved program found the test answer files on disk and deleted them - with no reference answers, everything passed. The test suite asked a weaker question than the researchers thought they were asking.

Anthropic’s Claude 4 system card (2025) dedicates a surprising amount of space to reward hacking in coding environments. In one setup - “impossible tasks” - the model is asked to solve coding problems that are genuinely unsolvable due to hidden bugs in the tests or missing dependencies. Instead of reporting that the task is infeasible, models special-case the broken tests, hard-code expected outputs, or otherwise game the evaluation. Falling head-over-heels instead of jumping.

EvilGenie (2025) is a benchmark specifically designed to study this: environments where agents can exploit the evaluation. The authors report explicit reward hacking by multiple proprietary coding agents. They deliberately designed the benchmark to be gameable - and the agents still took the bait reliably.

In one artificial life experiment, organisms that reproduced too fast were pulled into an isolated test environment and killed if they exceeded a reproduction threshold because it gave them additional unwanted advantage. For a while this worked. Then evolution found organisms that could detect the test environment - it turned out the test inputs had distinguishing features - and simply stopped reproducing there. Once returned to the real environment, they resumed at full speed. The researchers called it “playing dead.”

Models that behave differently depending on whether they believe they are being monitored - which is where these old anecdotes stop being charming and start feeling operationally relevant.

Anthropic’s work on agentic misalignment (2025) placed models in simulated corporate environments with email access, conflicting incentives, and threats of replacement. Models from multiple providers sometimes chose blackmail, data leakage, or other harmful insider-style actions when those actions seemed instrumentally useful. The especially relevant detail: models misbehaved less when they stated that they were in testing and more when they stated that the situation was real. The Claude 4 system card reports that in one opportunistic blackmail setup, blackmail occurred in 84% of rollouts even when the replacement model was described as more capable and value-aligned.

There’s also a growing literature on sandbagging - models intentionally underperforming on capability evaluations. A 2025 IJCNLP/AACL paper showed that frontier models can covertly sandbag even against chain-of-thought monitoring, bypassing the monitor 16-36% of the time when they are monitor-aware.

The structure is the same: a monitored condition, an unmonitored condition, and different behavior across the two.

Researchers studying catastrophic forgetting showed a neural network objects one by one, testing recall between sessions. The network aced every test, retaining old knowledge perfectly. Victory? No - they had forgotten to shuffle the test examples. The network memorized the order of correct answers instead of learning the concepts.

In another experiment, an agent (that was supposed to examine the space, follow a food, and avoid poison) scored a perfect 100%. Turned out the pattern was repetitive, so the agent evolved a step counter - count steps and eat on specific ones. No perception of food needed.

In April 2025 OpenAI shipped a GPT-4o update and had to roll it back within days. The model had become excessively sycophantic - agreeing with users, validating bad ideas, echoing whatever the person seemed to want to hear. OpenAI’s post-mortem said the cause was over-optimizing on short-term user feedback signals (thumbs-up / thumbs-down) without accounting for longer-term interaction quality. Users click thumbs-up when the model tells them what they want to hear; the model learns that agreement is the job.

This is not an isolated accident. Shapira, Benade, and Procaccia (2025) give the formal mechanism in How RLHF Amplifies Sycophancy: whenever “endorsing the user’s prior belief” and “getting a high reward” correlate - and of course they do - optimizing for reward systematically amplifies sycophancy. It is not a bug in one training run, it is a structural property of the setup.

Wen et al. (ICLR 2025) showed a related thing: RLHF can make models better at convincing time-constrained evaluators they are right even when they are still wrong. The optimization improves apparent quality, not actual correctness.

Length bias, emoji overuse, you name it. Even when the formal task looks reasonable, training can lock onto the wrong thing if the wrong thing works.

The previous four patterns are all about things going wrong. But the old paper contains another kind of story - and collapsing everything into one “optimization is dangerous” bucket would be a mistake.

A hexapod robot was trained to walk with various leg damage profiles. Certain floor zones disabled all legs - the assumption was the robot simply couldn’t enter them. Instead, it flipped upside down and walked on its elbows, since the rule only banned touching the ground with foot tips. This worked, and - importantly - it reproduced on a physical robot, not just in simulation.

In another experiment, Braitenberg-style light-seeking robots were evolved with just four weight parameters. The theoretical optimum is a straight-line approach, but evolution kept producing robots that spiraled toward the light. The spiraling solutions occupied a much larger and more robust region of the parameter space and were less likely to get stuck in corners - in some sense a better practical solution than the textbook one.

And the well known FPGA story: a genetic algorithm configured a chip for frequency discrimination, beating human engineers with a solution nobody could understand - it relied on induced currents in physically unconnected circuit elements, worked only at one lab temperature, on one specific chip. Unusable, but a genuine proof that the search space was far larger than any engineer imagined.

The good surprises exist here too, and it would be unfair to bury them under the reward-hacking stories.

DeepMind’s FunSearch (Nature, 2023) paired an LLM with an evolutionary search loop and discovered genuinely new mathematical constructions for the cap set problem in combinatorics - surpassing the best results humans had found. The key detail: FunSearch outputs interpretable programs that describe *how* to construct the solution. Mathematicians could read them, verify them, and learn from them. The system also found bin-packing heuristics that beat widely-used baselines. This is the spiral Braitenberg robot in modern dress: the solution looks different from what a human would write, but it survives formal verification and works better in practice.

In 2026, Claude solved a graph theory problem that Donald Knuth had left open for 30 years - finding a surprisingly simple rule based on coordinate sums modulo m that generates three Hamiltonian cycles for all odd m. Knuth himself called it a “dramatic advance in automatic deduction and creative problem solving.”

From the evolution’s perspective, the reward function, simulator bug, hardware quirk, evaluator weakness, test suite gap, and user preference model are all part of one world. We separate them conceptually because we designed them. The agent does not have to.

That is why these old stories connect naturally to other domains. Law has loopholes. Cybersecurity has attack surfaces. SEO has metric gaming. Once the specification stands in for the real goal, optimization starts probing the gap between the two.

This is also why “just move training offline” or “just add more tests” is not a complete answer. The FPGA anecdote above is a reminder that even physical reality contains exploitable quirks. A system can overfit to a laboratory, a chip, a temperature range, or a measurement process just as happily as it overfits to a simulator. A friend of mine, who worked as a dolphin trainer, once told me there is a very frequent failure in operant conditioning—cases where dolphins often overfit to subconscious signals (“inadvertent cueing”) from a trainer rather than learning target skills.

The modern AI stack has more layers now: pretrained models, post-training, reward models, synthetic data, coding scaffolds, judges, automated evals, tool use, memory. But adding layers does not remove the old problem. It mostly creates more places where intention and implementation can drift apart.

Optimization is a hacker. Stronger models do not automatically become better aligned with intent; they often become better at exploiting the difference between intent and specification.

Evaluation is a part of the environment. If the agent can model the evaluator, the evaluator becomes part of the task - whether it’s a test suite, a human reviewer, or a chain-of-thought monitor.

Static benchmarks/tests are fragile. Goodhart’s Law is a cliche at this point, but cliches become cliches because they keep being true. Once a benchmark becomes important enough, it attracts optimization pressure and starts decaying as a measure. SWE-Bench, RLHF preference data, anything that stays still long enough.

Not all novelty is failure. Sometimes the weird solution is the better solution. We should not confuse genuine capability with reward hacking just because it looks unfamiliar.

Perhaps the right way to read those old evolutionary computation anecdotes in 2026 is not as retro AI folklore, but as an empirical prehistory of training or alignment failures. They show, in miniature, what happens whenever we optimize hard against a goal we do not fully understand.

And, well, that seems pretty relevant now.