The Looming AI Crash, False Velocity and Reality. Vol 3 - The Software Problem

GenAI and LLM’s are a huge step forward in technology and they are changing the way we work. However, they are a not transformative technology breakthrough that will change the world like the industrial revolution did. Those ideas are pure hype. A growing consensus of researchers and experts think that LLM’s (Large Language Models) are no longer scaling exponentially. Growth is happening but without exponential growth the current training methods won’t yield AGI (Advanced General Intelligence). It’s also clear that LLM’s fail massively when they go outside their training, as Apple’s resent paper says. The majority of benchmarks used to test new models only test with their training data. As soon as they are asked to give answers that fall outside their training, they fail. They are great at predicting text, but they cannot reason. This is partially due to the fact that they don’t contain working models of the world as some other AI systems do. For example they do not and can not contain a physics engine. They can’t be trained to model the real world and give us truthful answers. If you present them with a novel question, they fail. All the GenAI companies are doing the same thing. They all train their models the same way. They are focused on making larger and larger models. The models are training on content created by people. So far, LLM’s can’t train themselves and the frontier models have already consumed most of human knowledge and experience. LLM’s cannot reason, they cannot accurately extrapolate on ideas outside of their training and they hallucinate.

LLM’s are a subset of Deep Learning. Deep Learning is a subset of Machine Learning. Machine learning is a subset of AI, a field of research that goes back to the 1950’s. Photo recognition is a good use case to help us understand how this works. Deep learning models are trained to distinguish between categories of images, photos of dogs and cats for example. They use neural networks, which is software that emulates our brains, they need massive amounts of training data, pics of cats and dogs that are each labeled by humans. We submit a picture of a dog as an input and ask the model is it a cat or a dog as an output. Then the model is told if it’s right or wrong, this process continues until it has high accuracy. Each layer does a pass on an inputed image and discovers something about it, until, layer by layer, the photo is identified and we get the output: Dog. What happens inside these models is what researchers call a black box. We don’t understand how deep learning models find their answers, between the time when you submit an image for identification and an output is generated, we don’t know what happens. No one knows the specific effect training has on the model, nor how the model decides on output. In the narrow use case of either/or photo identification, hallucinations are very low, making the training reliable, but when we use general training set like an LLM does, the rate of hallucination is about 10% because the model is guessing on the likely answer. A hallucination is bullshit answer that an LLM tries to present as genuine. Chatbots like ChatGPT hallucinate because their output is based on what is likely, not what is true. LLM’s also exhibit gender and racial bias, likely because of their training, as we are biased and it uses our knowledge and experience. Look at facial recognition. GenAI systems are much more likely to identify men with light skin accurately. That’s two different ways that LLM’s prove problematic. What would you do with employees who were frequently wrong and racist?

LLM’s have exponentially more layers than deep learning and require an unbelievably large amount of parallel processing. Like the technology that came before it, the heart of an LLM is a black box. No one can see into it, no one understands what processes lead it to its outputs. This makes training messy. Imagine if you had a machine in your kitchen that makes food, but you can’t see inside of it, it’s just a box, it doesn’t make any noise, tell it what sort of food you want, put in the ingredients and you get food. But you have no idea how it’s made. The food maker is trained on everything we know about making food, but if it needs to be adjusted or improved, we can only guess on what new data to give it, and the results will it will fail 10% of the time. No one really understands how the food is made.

In the early days of ChatGPT, back when it was a non profit, the big tech companies and venture capitalists decided to place a huge bet on LLM’s as the way AI would get to AGI and then ASI. Recall that these are the essential benchmarks that fulfill the promise of GenAI. Advanced General Intelligence promises to create broad PHD level expertise that can replace knowledge workers. Advanced Super Intelligence promises god like powers of intelligence, so advanced, that we can no longer comprehend them. To be fair, the initial results and progress were impressive, certainly they warranted some investment. But there was no evidence that LLMs would succeed. Making bets is what venture capitalists do, and it’s rare for them to put so much money on one bet. Over one trillion dollars so far. I’ve described this as a run away train, barreling down the tracks, needing to increase speed perpetually, this is a problem. All GenAI companies are pursuing more or less the same training methods, using the same hardware and very similar LLMs. They are not experimenting, or looking at alternatives. As good science dictates. Why? Because it would cause them to slow down.

We can clearly see that the growth curve of LLMs (Large Language Models) has flattened. In the early days of OpenAI, there was exponential growth, and the hope was, that scaling laws would drive exponential growth of the models and sooner or later they would become as smart as a person, then it’s AGI, as smart as a person with multiple advanced degrees. ChatGPT versions 1-3, had exponential gains, fair gains were had in 4 and then things get flat after that and it’s the same for all LLM’s regardless of who is working on them, where they are located. A 2024 paper, “Breaking Myths in LLM scaling and emergent abilities with a comprehensive statistical analysis,” published in ScienceDirect concluded, “Our results do not provide clear support for linear scaling, emergent abilities, or the effectiveness of certain training methods reported in prior studies.” It’s been two years since LLM’s have had exponential growth, yet investors keep piling on the money, and massive datacenter continue to be built.

In a 2025 blog post Sam Altman claimed, “The intelligence of an AI model roughly equals the log of the resources used to train and run it. These resources are chiefly training compute, data, and inference compute. It appears that you can spend arbitrary amounts of money and get continuous and predictable gains; the scaling laws that predict this are accurate over many orders of magnitude.” He was wrong. Gary Markus and AI researcher and NYU professor, concluded that LLM’s had hit a wall back in 2022. And the rest of the machine learning community is catching up. At the 2025 NeurIPS, the industries largest conference, there was a growing consensus that LLM training will not scale. There is no scaling law that applies to LLMs. Even Moores law hit a wall three years ago to do the limitations of quantum physics.

Nearly all AI companies are only pursuing LLM’s as their road to profitability. They are hoping that they will be able to overcome the limitations of LLM’s with blind faith. The idea that AI is unstoppable is total propaganda, as is the idea that AI will take away our jobs. There is no evidence for either of these outcomes. GenAI is closer to the SoundBlaster, Talking Parrot, published in 1990 than it is to AGI. Researchers predicted LLMs would hit a wall back in 2022, but by then, AI companies had overcommitted, investors had over-invested and shifting strategy meant they could not be profitable before debts would come due. Instead of slowing down to pursue new research, they just kept pumping in more money and praying for results.

A former Facebook researcher and winner of the Turing Award, Yann LeCun, left Facebook after predicting the LLM’s were a dead end to form a new company that’s exploring other avenues of research and received the largest funding of any startup in the EU, $1.7B. This is the future of AI, trying to find new models and methodologies that might get us to AGI and ASI. GenAI companies are beating a dead horse. They are going to fail because of their laser like focus on a single solution, their over investment in failing technology and too tight timelines to profit.

There will be no winners of the LLM race, not as long as the finish line is AGI and/or ASI. LLMs have lost that race, they can’t get to the finish line. The exponential gains are long gone and the future of AI is based on research and finding new fields or research to pursue. The new goal for LLM’s should be to focus on what they can actually do, and optimize for that. Not taking human jobs, but making us better at our jobs. They should augment us, do the boring, repetitive work, help us automate workflows, do translation, the boring parts of coding and the like. That’s a race they can win, sort of.

Because of the trillion dollars invested so far, if we redefine the race as I suggest, the GenAI bubble will burst. Without at least achieving AGI, they can’t generate enough revenue to break even, never mind justify their investment.