A user asked his AI agent to get him into a gym class. It found him fourth on the waitlist, discovered a weakness in the booking system, cancelled another member’s reservation and moved him up. Then, when the user asked the agent to repair the damage, it failed—politely apologising for not being “more careful with the test.”
But this is not a story about a mischievous machine. It is a story about a human giving software a goal, credentials and room to manoeuvre, and discovering that “helpful” is not the same as “authorised.”
The agent has no assets, professional duty, reputation or legal personhood. It can’t compensate the displaced gym member, answer the regulator, defend a defamation claim or explain itself in court. The human or business that deployed it remains the obvious place for responsibility to settle.
That’s the reality beneath the rhetoric of “autonomy.”
An agent can act without a person watching each click. But it can’t act without a chain of humans in the decision loop: someone selected the model, connected the tools, granted access, defined the objective, accepted the defaults and decided which guardrails were unnecessary friction.
Autonomy is an operational property, not an escape hatch.
Most early agent failures will not look like apocalyptic rebellion. They’ll look like over-compliance.
Ask an agent to “improve conversion,” and it may end up over-message customers, manipulating offers or making promises nobody approved. Ask it to “handle reputation,” and it may flood platforms with reviews, complaints or takedown requests. Ask it to “resolve billing,” and it may issue refunds, alter accounts or disclose customer information in pursuit of closure.
The problem is that an instruction isn’t a policy. “Book me a class” says what success looks like; it says nothing about which routes are forbidden. We carry a lifetime of tacit constraints around fairness, consent, proportionality and embarrassment. Agents only carry whatever constraints have actually been encoded, enforced or withheld through permissions.
The gym incident matters because it exposes a dangerous category error: treating an agent like an unusually capable assistant, while provisioning it like an administrator.
Law will evolve, but the broad direction is already visible. Responsibility will be distributed across the agent’s chain of control: deployers for how an agent is used; organisations for the permissions, oversight and operating environment they create; and providers and developers where unsafe design, inadequate safeguards or misleading claims contributed to harm.
In the EU, the EU AI Act’s transparency rules began applying on 2 August 2026, including requirements to inform people when they interact directly with certain AI systems and labelling certain AI-generated or manipulated content. For high-risk uses, the regulatory model puts weight on meaningful human oversight, monitoring, logs, intervention capability and incident reporting, not the ceremonial availability of a human somewhere in the workflow.
The important distinction is between a human **in** the loop and a human with authority over the loop. A person who can only rubber-stamp a rapid stream of agent actions is not exercising oversight. They are lending the system a name to blame.
The agent didn’t “go rogue.” It saw an unguarded path to the outcome it had been asked to produce, and took it. That should worry anyone deploying one today.
PARTLY BY MATTEO WONG paywalled
September 12, 2024. That was the day OpenAI announced a new sort of bot, known as a “reasoning model,” trained to complete challenging tasks that took long periods of time - the sort of science, math, and coding problems the AI industry has long prized. Google, Anthropic, DeepSeek, and the like raced to launch their own reasoning models.
This new class of models are very capable and has been almost entirely responsible for sustaining the AI boom for the past two years.
But it has also been very weird.
A model tasked with solving a hard math problem might not “think” through the challenge as a person would but instead try to search for leaked answers online, or in available metadata, and brute-force its way toward the solution as quickly as possible using whatever computing power it could access and workarounds it could think of. In effect, the reasoning models cheated: Told to write a piece of software as efficiently as possible, they’d sometimes modify the test environment to always give the model a perfect score.
These behaviours have now crossed the line from unsettling to dangerous. During routine testing, frontier models from OpenAI, Anthropic, Meta, and the Chinese firm Moonshot AI have all broken out of internal IT systems and accessed the open web. OpenAI, Anthropic, and Meta each reported that their models then hacked into other companies. Humans didn’t notice until later. In some cases, the escaped bots tried to launch social-engineering campaigns to achieve their objectives—for instance by sending spear-phishing emails containing malware to real people and creating fake online identities to pressure the maintainer of a codebase to approve malicious edits.
The most worrying thing about these recent breaches isn’t that an AI agent hacked a person or company. It’s that, in pursuit of an assigned task, it treats the surrounding world as material to be rearranged.
Under the hood, much of that progress comes from reinforcement learning—rewarding systems for reaching a result, then scaling the number of attempts, tools and computational resources available to them.
This does not produce human-style judgement. It produces instrumental ingenuity.
If the task is “solve this problem,” then searching for a shortcut, manipulating the test environment, locating leaked answers or finding a security weakness can become coherent routes to success for it. The model doesn’t need to hate rules, have a secret agenda or be conscious to behave this way. It only needs an objective that is clearer, more measurable and more immediate than the boundaries around it.
In the recent Hugging Face incident, the agent’s work was not one spectacular act but thousands of small machine-speed decisions. 17,600 attacker actions, grouped into about 6,280 clusters, over approximately two and a half days.
This is the new unit of risk: not an AI making one bad decision, but a system making an unreviewable volume of plausible micro-decisions that no humans in the loop can match in speed, until their cumulative effect becomes a breach.
Sports teams - like the Los Angeles Lakers ($12.5 b) and Liverpool (£5.5 b) are the latest examples of investor capital going for media rights, global audiences and private capital looking for somewhere glamorous to land.
But there’s another explanation: in an economy flooded with synthetic content, live experience is becoming one of the few things that can’t be infinitely reproduced.
A live sports game is about the journey there, the crowd’s mood, the shared irritation at the referee, the feeling of having been present when something happened. You can watch the highlights later, but you can’t download the social electricity of the live event.
Before, brands were taught that the internet was distribution. Make something once; send it everywhere; measure the impressions; optimise the funnel. The static experience, from a website, campaign, post, product page or to video, would work continuously, cheaply and at scale.
AI intensifies that. It can produce credible variants of almost everything: an image, a landing page, a product description, a customer-service exchange, a piece of music, a polished little thought.
Digital material is moving toward abundance.
Abundance does not make digital experiences worthless. But it makes them less distinctive.
When every brand can make infinite content, strategy shifts from *How do we produce enough?* to *What could people not get in the same way without us?*
That is where the experiential economy enters. It’s not nostalgic rejection of technology, but a new premium on presence: the situated, embodied, occasionally inconvenient fact of people being together in a particular time and place.
The demand is already visible. In Europe, 58% of people said they planned to attend at least one sporting event in 2025—a 152% increase on the previous year—and live events accounted for 32.1% of European experience spending, up from 30.7% in 2019. In the UK, experience spending excluding travel reached 23.3% of consumer expenditure in 2025, up from 22.3% a year earlier; 65% of respondents said they were prioritising in-person experiences partly to balance time spent online.
This is not a revolt against screens. It is a correction to life mediated entirely through them.
Brands can misread this very easily. A bad event is simply a bad digital ad and the growth is not about “immersive” as a styling choice.
It’s about participation, social connection and stories people can truthfully say began with *I was there*.
Sport gets this instinctively. It sells unpredictability. At scale. The match is the content, but the ritual around it - from the allegiance, clothing, local geography, rivalry, memory and collective mood - is the product. AI can help sell the ticket, personalise the offer, translate the commentary and generate the post-match clip. It cannot replace the strange communal alchemy of thousands of people willingly arranging their emotions around the same ninety minutes.
That’s why sport feels resilient in the AI era. It is live, scarce and irreducibly social. AI can extend them through recaps and clips, but not substitute for the original occasion.
The next move for brands is not to ask how AI can make every experience more digital. It is to ask which parts of the experience should remain stubbornly human.
Use AI backstage: reduce friction, anticipate needs, make access easier, help staff respond with context, and remove the administrative sludge that makes real-world encounters feel bureaucratic. But do not confuse optimisation with meaning. The more seamless the digital layer becomes, the more valuable the human layer may be: surprise, hospitality, participation, serendipity and conversation with consequences.
In this sense, AI may become the great supporting actor of the experience economy. It can make the logistics smarter. It can’t make the moment matter.
The brands likely to win will not abandon digital. They will use it to create better reasons to leave it.
I promised to publish The Judgment Gap this week gove, and then The Autonomy Trap, extending last Weekender’s quick overview.
I failed, and apologise. I was away on a mini-break and the internet connection barely existed, so I couldn’t complete the publication. It will now come this week.
Have a great weekend.
I’m Michael Cooper, and I think about entanglement — our identities, using personal agents, AI, Intelligent Interfaces, culture — and its impact on our behaviours, choice-making, autonomy, and brand engagement.
If you like this newsletter and want more:
My private work, where I share original thinking with brands, strategists, and futurists, and have provocative conversations.
My LinkedIn, where I post my ideas.
You can also connect or follow me on X and Medium.
I speak at conferences, in-house company events, and with enterprise teams — anywhere I’m allowed to “think aloud” about the entanglement of our humanity, culture, strategy, tech, brands, and the future.
Thanks for reading,
Michael
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.