LLMs and Cognitive Surrender

Just a reminder that LLMs (AI) are not to be trusted. I recently came across a friend posting an “AI summary” declaring that there are no speed limits for bikes in QLD.

This is very much false and if you follow the “AI” you could end up with a very expensive fine.

An LLM just generates text which statistically looks like it should fit together. This means that it’s never just providing you with a copy of raw facts and it can, and will, generate things which sound plausible but are absolutely false. Remember, it is not a thinking machine, it can not lie or tell the truth, it just generates a statistically plausible response to your question. Do not personify LLMS.

This means you should treat everything generated by an llm as false until you’ve personally verified it. Use it as a way of generating a search to an authoritative document if you’re having trouble formulating the search, but don’t use what it generates as the answer.

As these things bleed into more and more of our lives and we see more and more cognitive surrender and more enshittified Internet, I’m becoming less and less convinced that there’s any value in LLMs. Also, LLMs are subject to poisoning attacks and are the target of state sponsored propaganda networks.

The recent leak of Claude code showed what code generated by LLM experts looks like and wow, do I not want to be anywhere near that code base. It’s an absolute shambles (with bugs which were very easy to spot by software engineers) and, if that’s what the very best people write, we’re in for a very rough ride in the realm of software quality.