I know everyone and their dog is writing about LLMs, and since I
refuse to use them</a> you probably shouldn't
listen to me, but I have recently solidified some of my thinking about them and
thought it might be useful to share. None of this makes any difference to the
disastrous ethical position of LLMs, but it has helped me understand them a bit
more, and fear them a bit less.</p>
My PhD was in
using artificial evolution to design neural networks</a>
that performed cognition-like tasks in simulated robots.</p>
I found that artificial evolution can produce stunning results: behaviour and
patterns "emerged" from my experiments that I felt I had not designed in to the
systems. This prepared me well for the "AI vertigo" that many of us feel when we
see the output of modern AI systems.</p>
What artificial evolution does is allow us to explore overwhelmingly large
search spaces and find interesting islands. What this means in practice is that
we can represent any system (in my case, the neural network controlling a
simulated robot) by a list of numbers. Searching the space means changing the
numbers by small random amounts, and then evaluating the results, and repeating
many times over until we find combinations of those numbers that produce "good"
outcomes.</p>
To explore our space, we need a way to transform numbers into the system we are
interested in, and we need a way of judging what is "good".</p>
I quickly learned during my research that it matters enormously what the "shape"
of your search space is: essentially, if changing a single one of your numbers
completely changes the nature of the system, then artificial evolution has no
"slopes" to crawl up while it explores the space, and nothing interesting
evolves.</p>
In fact, I would say that the shape of the search space is the most important
factor in working in this area - more important than, say, the exact design of
the experiment. In the field, people talk loosely about this property as
"evolvability".</p>
Recently, Deepmind
achieved impressive results</a>
in Mathematics. My naive understanding of how their "Deep Think" (seriously?)
system works is that it generates potential lines of reasoning using an LLM and
then chooses which lines to pursue using additional systems.</p>
Understanding this work in the context of searching a space helps to connect it
to my own research: this system is exploring the unimaginably large space of
possible lines of mathematical reasoning and trying to find locations within
that space that are "good" in the sense that they represent valid arguments that
are relevant to specific mathematical problems.</p>
What LLMs bring to the table is that they change the shape of the space: instead
of picking the next word to try out randomly from a dictionary, the LLM presents
possible lines of argument that look "plausible" based on the statistical
properties of similar text included in its training set.</p>
The key property of this system is that it does not consist of an LLM working on
its own: LLMs essentially produce random plausible-looking output. The way it
works is that the LLM "smoothes" the search space for an optimisation algorithm,
making it easier to find good solutions.</p>
This framing (an LLM smoothing a search space while external systems do the
searching) helps explain why some areas have proved fruitful for "AI": for
example, if we use a highly-constrained programming language like Rust, and
combine it with lots of tests that constrain the behaviour, it is possible to
generate working programs. The LLM produces random plausible-looking code, and
the constraints allow us to judge what "good" means.</p>
This gives me a good frame for understanding LLMs: why do they "work", why are
they terrible, and what will happen longer term?</p>
Why do LLMs sometimes "work"? How can essentially random systems produce working
code or solve Maths problems? When we see that the LLMs themselves are
effective at producing "plausible" randomness that is then guided by other
systems, it's easy to see that, just as my experiments using artificial
evolution (based on searching using random numbers) produced results, these
systems can do the same: the guidance as to what is "good" provides meaning,
and LLMs make good solutions easier to find by providing a smoothed search
space.</p>
Why are LLMs terrible? The worst current uses of LLMs miss out the crucial
guidance part, and rely purely on the plausible randomness1</a></sup>. Of course, this
means the chatbots produce garbage and bring on AI psychosis by echoing what we
feed into them. Meanwhile, more guided systems produce "working" code or
artifacts that are unsustainably chaotic in their construction. This makes sense
if we see them as random noise that has been shaped by the constraints provided.
They are very much like the systems I worked on during my research, which
produce meaningful behaviours, but are very strange and alien when we examine
the details of how they work.</p>
The additional "supervised" training done on recent LLMs which shapes
their output to fit some particular criteria may be seen as providing some
external framework to the basic LLM, but it seems demonstrably clear that
systems trained in this way still produce dangerously incorrect and harmful
output. Only external, deterministic systems can constrain these systems to
produce output that fits some required criteria.</p>
</div>
In future (hopefully) LLMs will disappear into the background, feeding plausible
randomness into some more discerning search process. Scaling back our ambitions
for the LLM part of the system from all-knowing oracle to plausible
noise-generator may even allow us to scale back their size, providing some path
towards reducing their devastating climate impact and cost. If we choose to, we
could even stop stealing the creative work of the whole of humanity and train
models on data to which we have permission. I can dream.</p>
When we see LLMs as a better random number generator (for some definition of
"better"), we can see that once the hysteria is over, they won't be replacing
people any time soon.</p>
The hype surrounding LLMs, and the accompanying abandonment of ethics in the
entire technology industry, remain the most important concerns. But we can
understand LLMs a little better if we see them as generating "plausible noise":
they can be effective as components in a wider system that uses the noise as a
random input, and figures out the quality or value of what it sees using
well-understood, external, deterministic criteria.</p>
</a> I hate to say it, but I
think this take is about two years out of date. There's a lot more to it than
"guided randomness", and getting "garbage out" is becoming vanishingly rare
(which is not to say that the output is perfect).</p>
LLMs are not the same as human intelligence or reasoning - even if they give
a very good approximation of it (leading to the "AI Vertigo" you mentioned). But
let's separate the majority's tendency to anthropomorphise from their usefulness
as tools (and also separate - for now - the ethical, economic and other concerns
- which are definitely a big deal, but also a different topic).</p>
Can LLMs 100% replace all human coders right now? No? Does that mean they are
a passing fad that are probably more a distraction than something useful? Not at
all!</p>
What LLMs can do is write surprisingly high quality code (sometimes with a
bit of guidance) at an astonishing rate, and drawing on a body of knowledge that
no human has. It can still be used to build buggy balls of mud that fall over
when you push them. But a good developer driving, who is paying attention, can
build high quality software 10-100x the rate they could previously - and often
software that was not possible or feasible for them, previously.</p>
This certainly leads to local optimisation problems (right now we have a PR
volume crises!) and we're learning how to effectively use these tools - right as
they are changing under our feet. But this is a genuine watershed,
genie-out-of-the-bottle moment. It's not all upside (and you can argue very big
downsides) but there is no going back.</p>
</div>
I would argue that the noise has become more plausible over time, but has not
really changed in nature. I think we'll find out more over the next 12 months or
so.</p>
Meanwhile I want to be very cautious about saying things like "there is no
going back" about a technology that is causing significant harm to people and
climate. It serves the ethically-bankrupt billionaires driving the project, not
us normal people. Similarly, the "the last model was garbage, but this one is
unbelievable!" narrative.</p>
</div>
In terms of "signs of intelligence", or similar epitaphs (that are hard to
agree on even the definitions of) the "become more plausible" and "we'll find
out over the next 12 months or so" may be right (depending on those
definitions). That's why I focused on their usefulness as a tool - especially
for coding. These tools are acting as a huge multiplier _today_. Not everyone is
wielding them well, and the big problem is how we incorporate that into the
whole pipeline (we're currently creating bottlenecks elsewhere - especially with
the lowest effort use).</p>
Regarding "the last model was garbage, but this one is unbelievable". I don't
believe I have ever heard it quite so starkly. But we did say things that might
sound similar (e.g. "the new models are a huge step up from the previous ones,
and fix a lot of shortcomings") - mostly _last year_. Perhaps even earlier this
year. But most of this year has been characterised by a shift more towards
steady state - new models have incremental improvements. We're learning to use
them more effectively and build better (meta)tooling around them. More often
people are choosing to use _less_ powerful models because they are already "good
enough" but cheaper or faster. That's not to say there are no more step changes
coming up that _for some things_ might move from "not there yet" to "now we are
there". None of that negates the value we can derive _right now_.</p>
And the "no going back" is not something that I'm giddily proclaiming based
on my preferences! There is an incredible scale of downside to this that may
very well accelerate the end of mankind - at least as we know it! And that's on
multiple fronts! If there was any way we could campaign and convince a
representative proportion of our community (at least) to make a stand and stop
the whole thing, I would be right there! Even if you took out the interests of
the "ethically-bankrupt billionaires" I don't think much would change, though.
It might slow things down a bit (which would be welcome) and it might ease some
of the knock-on effects (costs of RAM and GPUs for example) - and even slow a
bit the appetite for new data centres - but only a bit. The problem is not just
some small group of corrupt elites. They are as much a symptom of the problem as
anything else! The problem is _us_! Collectively. We've been down this path many
times before. Every big revolution has brought with it change for the worse -
and more often than not it outweighs any good. There has _never_ been any going
back. This is an unstoppable force.</p>
That doesn't mean giving up and joining the other side. Each big revolution
has brought with it something good, too - and sometimes that _does_ outweigh the
bad. That's the one lever we may be able to pull. Can we _use_ this new, still
growing, technology to actually solve or mitigate some of the big problems -
including the ones it is creating? Can we accelerate discoveries and proving out
of new energy sources (or more efficient uses of what we have)? Can we find
cures for more diseases, faster? Can we direct the economic shifts that are
already accelerating so we end up in a post scarcity society where those
billionaires have no advantage ? And so on.</p>
All extremely optimistic, and arguably unrealistic. But, importantly,
suddenly more possible than ever before - and probably our only hope!</p>
Sorry, this turned out a bit longer (and existential) than I had expected. I
didn't have enough time to make it shorter, etc...</p>
I have thoughts :-)</p>
</div>
I appear to have accidentally joined the efficacy debate when I swore I would
stick to ethical acceptability.</p>
</div>
Phil: the reason why LLMs in 2026 are any "good" at code at all even on their
first try is precisely because of the process Andy is describing.</p>
All LLMs are "pre-trained" on copyrighted books etc, then "aligned" by going
through an iterative process in which low-paid humans (or another model you want
to "distill" from) repeatedly evaluate the new model's candidate answers and the
model is adjusted until the evaluations are higher. LLMs that can write code
are also set to write code during training, which is automatically tested and
the model adjusted until it has a higher probability of generating code that
passes the tests on the first try.</p>
Because this part of the testing can be fully automated, the models end up
better at writing code than coming up with answers to other things. (Seriously,
last week I asked Gemini on Google Search if there was a way to replace a
proprietary connector on a failing LED room light, the manufacturer's answer
when I eventually got through to them was "no" but Gemini had suggested buying a
battery light and taping it to the failed light; I had to point out that
wouldn't get the same lumen output.)</p>
Even though a coding model has been refined to increase the probability of
getting it right on the first try, it still frequently fails to do so and
therefore putting it in an external process to make it refine its answers still
works better. Many models are also trained such that if you give them some code
they previously generated and a compiler or test suite error, they have a high
probability of patching it to an improved version in the light of said error.
Sometimes it even works just to ask the model again "what's wrong with this code
you just wrote". That makes it relatively easy to improve the code output at
large energy expense just by wrapping the model in a harness that runs it
repeatedly.</p>
I still think it's cheaper to run experienced humans when quality really
matters. The fact that so often in the commercial world quality doesn't
actually matter because everything is supposed to be just a throwaway demo or
whatever, is a separate problem....</p>
</div>Background: the shape of search spaces</h2>
Deepmind's Maths results</h2>
Finding good solutions</h2>
This changes how I understand LLMs</h2>
Summary</h2>
Comments</h2></a></p>
Phil Nash
2026-07-22</span>
</h3></a>
Andy B
2026-07-23</span>
</h3></a>
Phil Nash
2026-07-23</span>
</h3></a>
Andy B
2026-07-23</span>
</h3></a>
Silas
2026-07-24</span>
</h3></a>
This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.
I know everyone and their dog is writing about LLMs, and since I refuse to use them you probably shouldn't listen to me, but I have recently solidified some of my thinking about them and thought it might be useful to share. None of this makes any difference to the disastrous ethical position of LLMs, but it has helped me understand them a bit more, and fear them a bit less. Background: the…
1</sup>
</a>
</a>
</a>
</a>
</a>
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.