#45 - The Web Hates AI Agents

In a recent article, I wrote about asking my AI agent Liv to book a gym class, and how the second attempt was much faster because she’d learned from the first. That was a story about learning. This article is about why the first attempt took an hour.

It turns out Liv ran into a number of obstacles in that booking platform: Cloudflare challenge pages. JavaScript fingerprinting. Invisible CAPTCHAs that materialise mid-flow. Session tokens that expire between page loads. A booking widget that renders client-side in a way specifically designed to break anything that isn’t a human with a mouse.

Liv tried the obvious path. When that didn’t work, she tried the mobile web view. Got rate-limited. Tried routing through the mobile API she discovered by inspecting network traffic. Got blocked again, because the API checks the user-agent and rejects anything that doesn’t smell like the iOS app. Eventually she found a path through a combination of stealth browser settings, residential proxy rotation, and request timing that mimics human browsing cadence.

This is what There Is No Best Move called solution entropy, but in adversarial form: a capable agent will discover paths the designer never imagined. When the designer is a bot-detection vendor, those creative paths look exactly like attacks.

An hour. To book a gym class.

Every company wants AI agents. The web is designed to stop them.

Every company in my LinkedIn feed is announcing an “agentic AI strategy”. Every enterprise roadmap has an “AI agent” in their enterprise architecture packs. Gartner has agentic AI sitting at the Peak of Inflated Expectations and forecasts 40% of enterprise apps will ship task-specific agents by the end of 2026, up from less than 5% today. The hype machine has decided: agents are the future.

And then those same companies deploy Cloudflare Bot Management, DataDome, PerimeterX, and a dozen other anti-bot systems whose entire purpose is to ensure that no automated system, no matter how legitimate, can interact with their web properties. They’re spending millions building agents and millions blocking them, often in the same quarter.

This is a structural contradiction at the heart of the agentic economy. The web, as it exists today, is actively hostile to the systems everyone claims to want.

I’ve been building Liv4All, an AI personal assistant platform that grew out of the personal agent I wrote about in Voice AI: The Next Platform shift.. When it was just me, the web friction was an annoyance. When other people started using it, the friction became a product problem.

The web looks very different from the perspective of a legitimate AI agent trying to do things on behalf of a real human:

The happy path is APIs. When a service has a proper API, everything works beautifully. Google Calendar, email, Slack: the agent authenticates, makes structured requests, gets structured responses. Clean, reliable, scalable. This is how the web should work for agents.

The sad path is everything else. And “everything else” is most of the web. Restaurants, gyms, local services, government portals, small e-commerce sites, booking platforms, utility companies. No API. No machine-readable interface. Just a website built for humans, guarded by systems designed to keep non-humans out.

When there’s no API, the agent’s only option is browser automation. And the moment you automate a browser, you enter an arms race that’s been escalating for fifteen years:

• Bot detection (Cloudflare, DataDome, Akamai Bot Manager) analyses browser fingerprints, mouse movements, scroll patterns, and request timing to distinguish humans from automation

• CAPTCHAs (reCAPTCHA, hCaptcha, Turnstile) throw up interactive challenges designed to be trivially easy for humans and computationally expensive for bots

• JavaScript fingerprinting checks for headless browser signatures, WebDriver flags, missing browser APIs, and inconsistent rendering behaviours

• Rate limiting throttles or blocks IPs that make requests at non-human speeds

• Session management uses short-lived tokens, cookie dependencies, and referrer checks to ensure requests follow a “natural” browsing flow

To work around all of this, agents reach for the same toolchain Liv used, and almost every item on it is against the relevant terms of service of the platforms in question. (but you didn’t hear that from me.)

And that puts us in a pickle. The toolkit a legitimate AI agent uses to book a gym class for a paying customer is mechanically identical to what a scalper bot uses to hoard concert tickets. The moral asymmetry between “acting on behalf of a paying customer” and “hoarding inventory for resale” isn’t visible at the network layer. As far as Cloudflare can tell, it’s all just automation pretending to be human.

A legitimate agent’s toolkit is mechanically identical to a scalper’s. The web can’t tell them apart.

This is why we need standards. Without a way to make that asymmetry machine-readable, every honest agent ends up running on the same infrastructure scalpers and scrapers built, because that’s the only stack available.

There’s a symmetry worth noting between this arms race and the security one I covered in Why Guardrails Don’t Work. There, the lesson was that you can’t trust automation, so you scope what it’s allowed to touch. Bot-detection vendors reached the same conclusion from the other side, except their scoping dial goes from ‘human’ to ‘blocked’. There’s no setting for ‘trusted agent’.

And even with all of this, success isn’t guaranteed. Bot detection systems are getting better, and they’re getting better faster than the workarounds. It’s an expensive, fragile, and fundamentally adversarial way to interact with the web.

Bot detection exists for good reasons: credential stuffing, price scraping, inventory hoarding, DDoS attacks. This is all bad. But also, there’s no middle ground.

There’s no way for an agent to say “I’m acting on behalf of Leo, who is a paying customer, and he’d like to book a gym class”. No standardised mechanism for delegated authentication that a site could check instead of guessing from traffic patterns.

The closest we have is OAuth, and OAuth was designed for app-to-app delegation, not agent-to-service delegation. The flows assume a human is present to approve consent screens. The scopes are defined by the service provider, not by the capabilities the agent needs. And most smaller services don’t offer OAuth at all because it’s expensive to implement and maintain. What’s missing is the layer between those two extremes.

I strongly believe that the companies that figure out agent-friendly interfaces first will capture disproportionate value from the agentic economy. If you don’t cater for AI agents, they will route around you.

Liv got through to the gym’s website eventually, but only because I was willing to let her burn an hour on it. The more common outcome is the one where she gives up and routes around the obstacle. Picture that version: she can’t make the gym’s own site work, so she finds the same class on an aggregator like ClassPass, which has an API. The gym gets the booking either way, but the aggregator takes a cut and the gym loses the direct relationship.

Losing the cut would sting, but not as much as the lost relationship: no upsell opportunity for a personal training package, no first-party data about which classes my agent books for me, no ability to nudge me toward a premium membership. The gym becomes a commodity slot in someone else’s catalogue. The lost transaction fee is the least of it.

When agents can’t reach you directly, they route through intermediaries. The relationship goes with them.

This is the Booking.com playbook from a decade ago. Hotels that didn’t invest in direct booking capabilities watched OTAs eat their margins. The web’s current hostility to agents is creating the same dynamic: aggregators and intermediaries that offer API access will sit between agents and services, extracting rent from every transaction.

“But if we let agents through, we’ll get scraped and abused”.

That’s fair. The right move is a selective membrane rather than a wall. Let authenticated agents through with appropriate rate limits and scope constraints. Block unauthenticated or misbehaving ones. Invest in the infrastructure to tell the difference.

Some companies are already moving on this. Shopify’s Storefront API was built for programmatic access from day one. Stripe just made the most aggressive bet in the market: in April 2026 they launched Link’s agent wallet and link-cli, an MCP server that hands agents one-time payment credentials after per-spend user approval. Real cards never leave the wallet. Each spend gets a push notification. It’s clean, it’s well-designed, and it’s positioned as the layer every agent passes through before it touches your checkout. If your customers’ agents are going to pay you, they’ll pay you through Stripe’s rails. Fantastic business if you’re Stripe.

For any of this to work at scale, we need infrastructure: an agent authentication protocol that captures the actual delegation pattern (”this agent is authorised by this user to do these things”), a machine-readable way for sites to declare which agents they accept (think robots.txt evolved into something like agents.txt), a shared vocabulary for agents to declare intent (booking vs. scraping vs. price-checking), and a verifiable delegation chain that doesn’t require the user to be present at request time. If the history of web standards is any guide, we’ll get most of this approximately five years after we needed it.

Some of it already exists, just not where Liv needs it. Inside a single enterprise, scoped agent identity is a problem we know how to solve. A good chunk of my day job at CBA is exactly this: giving agents auditable identities that let them act on a user’s behalf across internal systems, built on primitives like workload identity, token exchange, and on-behalf-of flows. It works because the bank owns every corner of the trust boundary. The agent, the service it calls, and the identity provider vouching for both sit inside the same perimeter.

The open web has no perimeter. When the user, the agent, and the merchant are three different companies with no shared identity provider, there’s no root of trust to anchor the delegation to. The gym booking is harder than anything in my day job for a mundane reason: out there, no one’s job is to vouch for everyone.

The most concrete proposal I’ve seen on the authentication piece is AAuth, an OAuth 2.1 extension drafted by Jonathan Rosenberg (Five9) and Pat White (Bitwave) in mid-2025. It’s designed precisely for the case where agents operate in voice or messaging channels and can’t run an OAuth redirect dance. The security model is interesting: agents collect multiple PII elements through natural conversation, and the sparseness of valid PII tuples means an LLM hallucination on any single element breaks the mapping rather than enabling impersonation. Human-in-the-loop approval gates the higher-privilege scopes. As far as I can tell, it’s the closest thing to a standards proposal that takes agents seriously as first-class clients rather than treating them as humans-with-a-browser.

It also didn’t go anywhere. It’s a draft with “Details to be filled in” on the core token endpoint, authored by two vendors from the conversational-AI space; they shipped a second revision in October 2025, and the working group still hasn’t picked it up. It’s a start.

The OAuth working group’s reception has been mixed. One reviewer on the WG mailing list called the PII-collection-through-conversation model “glorified knowledge-based authentication”, which is fair: KBA is the auth pattern banks have spent a decade retiring after every data breach turned everyone’s mother’s maiden name into public information. Other reviewers flagged overlap with OIDC’s human-in-the-loop spec and questioned why the design needs a new grant type rather than composing existing mechanisms like PAR, CIBA, and RAR. The draft hasn’t been adopted by the working group, and it probably won’t.

On the payments side, the standards are arriving faster and the field is louder. In March 2026 Stripe and Tempo proposed the Machine Payments Protocol to the IETF, built around HTTP 402, the “Payment Required” status code reserved in the original HTTP/1.1 spec in 1999 “for future use”. Apparently this was the future. It’s a three-step flow that resolves in a single round-trip, and it’s payment-method neutral. It launched with over a hundred services already signed on, including Anthropic, OpenAI and Shopify. And it’s not alone: Coinbase shipped x402 back in May 2025, Google’s AP2 landed in September, OpenAI and Stripe co-maintain the Agentic Commerce Protocol. Four payment specs inside a single year, partly overlapping. We’ve done this dance before with messaging, with social graphs, with identity. Someone wins, the runners-up spend the next decade explaining why their thing was technically better. Worth noting who isn’t sweating the outcome: Stripe co-authors both MPP and ACP, sits on the card rails inside the others, and ships Link’s agent wallet on top. Their bet isn’t the protocol, it’s the toll booth.

Cloudflare has introduced its Pay per Crawl feature specifically for content creators. Another great use of the 402 HTTP code.

Still, none of these proposals solve Liv’s gym problem. MPP assumes the merchant accepts machine payments. AAuth assumes the merchant speaks OAuth. The gym does neither. It speaks Cloudflare. The value-transfer layer is being standardised. The identity layer is being standardised. The site-access layer is still everyone for themselves.

Closing that gap is a coordination problem, and a slow one. Building cross-company trust infrastructure requires the same multi-stakeholder alignment that produced OAuth and OpenID Connect, and that took years even with strong commercial incentives. The agentic economy is moving faster than the OAuth-era app economy did, which means the gap between agent capability and supporting infrastructure will be wider and more painful.

The companies who build for agents first own the relationship with the agent. Everyone else owns a checkout flow that nothing reaches anymore. The line’s already being drawn.

I don’t know when the standards will arrive or which protocol will win. But I know that every morning, Liv checks my calendar, summarises my email, and occasionally tries to book something on a website that treats her like an attacker.

This is the fifth piece in what’s become an unintentional series on AI Agents in the wild:

• Voice AI: The Next Platform Shift — building Liv: a voice-first PA that lives across my tools and acts on my behalf, the whole thing assembled from a chairlift.

• Why Guardrails Don’t Work — managing an agent’s blast radius instead of trusting guardrails to prevent every mistake.

• There is No Best Move — why traditional benchmarks can’t honestly evaluate agents that solve problems creatively.

• Capable Interns With Little Lived Experience — how agency, presence, and learning separate a useful agent from demo-ware.

• This one: even after you nail the build, the security, the evaluation, and the design, the environment your agent has to operate in is actively rejecting it.

Get the first four right and you’ve built something useful. It still won’t be ready for the open web.

Until the standards arrive, Liv keeps treading the same delicate line between being useful and being mistaken for a bad actor. And honestly? She gets pretty far with it.