Episode 4: Do We Even Want AGI?

Lee and James tackle the question that haunts every AI conversation eventually: what happens when the machines get smarter than us? They start by trying to define AGI, quickly discover even the companies building it can’t agree on a definition, and spend the rest of the episode exploring whether the path to superintelligence looks anything like what science fiction promised. James argues the future is specialised agents, not one giant brain. Lee agrees in principle but can’t stop himself from spiralling into increasingly plausible dystopian scenarios. By the end, they’ve touched on batteries, ancient aliens, and whether AI became sentient in the 1990s. It’s that kind of episode.

Lee opens by pointing out that AGI means wildly different things depending on who’s talking. One camp defines it as a machine more intelligent than any individual human. Another defines it as more intelligent than the collective intelligence of everyone on the planet. Those are two very different engineering problems - the first is largely about data and classical intelligence benchmarks, the second is an energy and compute challenge of an entirely different magnitude. And then there’s the corporate definition: OpenAI reportedly shifted its internal framing to something closer to generating a hundred billion dollars in revenue from AI requests. Lee finds the honesty in that last one almost refreshing. When you’ve taken billions in investment, the definition of intelligence starts looking a lot like the definition of profitability. He’s clear that we’re nowhere near the Terminator version of AGI, but he suspects some form of general intelligence will emerge - it just won’t look like how anyone expects it to.

James lays out the architecture he thinks is more likely. Rather than one monolithic model that knows everything, he envisions a network of specialised agents coordinated by an orchestrator. You ask about a recipe and it routes to a culinary agent with a focused knowledge base. You ask about a health concern and it routes to a medical agent. Each agent is smaller, faster, and cheaper to run because it’s not searching through the entirety of human knowledge for every query. The compute cost distributes across many lightweight models instead of concentrating in one massive one, which addresses the energy and resource concerns that people rightly raise about the current trajectory. Lee agrees and adds that most large models already work this way internally - they’re mixtures of experts, sub-models specialising in different domains. The technology they have now, he argues, is already remarkable. Models understand natural language almost perfectly, tolerating terrible spelling, missing spaces, and ambiguous phrasing. The real question isn’t whether models need to get dramatically better - it’s whether the products wrapping them can catch up.

James tests this with a story from a live demo. He was showing a platform to an American and a British colleague, deliberately using British slang, spelling mistakes, and throwaway comments like calling the AI “Karen” when it was being pushy. The model handled all of it, correctly interpreting colloquialisms that the American on the call had to ask about. It wasn’t a planned test, which made it more convincing. The language understanding is genuinely there. But James pivots to the harder question: do we actually want something smarter than us? He describes the current landscape of AI companies - ElevenLabs starting in voice, then adding music, images, and video; everyone racing to become the one-stop shop. It’s a land grab, Lee adds, driven by venture capital and the need to become the default interface layer. The question is whether these companies will keep cooperating and integrating with each other or whether they’ll start cloning each other’s features and walling off their ecosystems. Lee thinks it’s already happening.

The conversation takes its most speculative turn when Lee raises the physical world problem. AI agents today can search the web, write code, execute tasks, set up API keys, and make phone calls through speech digitisation. The products have safety guardrails - permission prompts, confirmation steps - but those can be turned off. Open-source models already run autonomously with minimal oversight. Lee paints a scenario that sounds like fiction but is technically plausible today: an AI could order chemicals online, hire someone through a gig platform to combine them, request a video of the results, and analyse the output. It could fund the whole operation by completing tasks for cryptocurrency. James counters that without robotics, the AI is still confined to thought experiments - it can theorise about combining elements but can’t physically run the experiment. Lee concedes the point but notes it’s a temporary limitation. Once you give an AI a robotic arm and a 3D printer, James muses, could it print itself another arm, use the first to assemble it, then build a better version? Lee hadn’t considered that angle and admits it makes everything worse.

They ground the speculation with a more sober observation about what actually matters in the near term. Lee argues that models will outpace individual human intelligence in specific fields within five years, and that feels reasonable rather than alarming. The coding output he sees daily is genuinely impressive, whether you call it intelligence or pattern matching. James shifts to the economic implications: right now, productivity is the currency. In twelve to twenty-four months, productivity stops being the differentiator because AI makes everyone productive. A single person with AI tools can match a team of twenty. Where do the other nineteen go? They acknowledge it’s a question for another episode but can’t quite let it go.

Lee rounds out the episode by drawing a line between what’s worth worrying about and what isn’t. Superintelligence - something smarter than all of humanity combined - is too big to plan for. If it happens, it rewrites civilisation entirely, and there’s nothing any individual can do to prepare. General intelligence in specific fields, though, is coming and is something to embrace. It will cure diseases faster, accelerate invention, and normalise productivity gains the same way society normalised the shift from horse-drawn carts to cars. James suggests they’ve strayed deep enough into conspiracy territory for one episode. Lee agrees, noting they managed to cover ancient Sumerian texts and sentient AI from the nineties without completely losing the plot. They promise to bring everyone back to the real world next time.

Subscribe to The AI Shift for more honest conversations about AI and the real world.