Agents of Change, Indeed

In Sintra, on the last day of June, the Bank of England’s Deputy Governor for Financial Stability told the ECB’s own forum that regulators should look at building “circuit breakers or kill switches that would limit or stop trading market-wide if faulty AI models cause market meltdown.”

A kill switch is not routine plumbing. It is the thing you install once you’ve stopped trusting the driver.

Sarah Breeden was plain about why. “Our frameworks were not built to contemplate autonomous agents,” she said, “and relying on a human in the loop for all agent actions is unlikely to be realistic.”

Translated: the system standing between your pension and a machine-driven crash has no rulebook for the machines already inside it. Nobody in the room gasped. That, in itself, is worth noting.

What almost nobody noted, in the month of coverage that followed, is that the Bank has spent the last decade building the exact apparatus meant to catch this sort of thing before a deputy governor has to improvise a metaphor about kill switches at a conference in Portugal.

It’s called CBEST (Critical National Infrastructure Banking Supervision and Evaluation Testing). And CBEST’s own findings suggest the emergency brake isn’t the most urgent problem.

“AI in finance” conjures the wrong picture, so it’s worth being precise. A chatbot drafts your email and waits politely to be asked again. An agent keeps going, it executes the trade, places the order, moves the money, settles it, all without pausing for a human to nod along at each step. Think of software authorised not merely to recommend a holiday, but to book it, pay for it, and adjust the rest of your finances around it without further approval.

Breeden isn’t worried about faster spreadsheets. She is describing several thousand unsupervised junior traders who never sleep, never second-guess a position, and cannot be talked down — because there is no longer anyone standing over their shoulder to do the talking.

And the moment for that worry has already arrived, not approached. Her own cited figure, 52 percent of finance firms are already experimenting with agentic AI. Not a forecast for next year’s planning cycle. A majority, already converted, ahead of the rulebook, ahead of the kill switch, ahead of pretty much everything except the marketing material.

She added a second observation that deserved to travel further than the first. The length of task the most capable models could reliably complete has been doubling on a roughly seven-month cycle, more recent measurements by evaluation bodies such as METR (Model Evaluation and Threat Research) show that period compressing toward four months.

“An already exponential increase in capability appears to be accelerating,” Breeden said — the sort of sentence a central banker permits herself once, quietly, before moving briskly to the next slide, because dwelling on it does not help anyone’s afternoon.

MPs got there first, and said it rather less politely. In January, Treasury Committee chair Meg Hillier warned the sector wasn’t prepared for a major AI-related incident, and the committee described the Bank and FCA’s posture as a “wait-and-see approach” exposing the public to “potential serious harm.”

The Bank and FCA agreed, in April, to take steps. Six months between a cross-party committee saying it in writing and a deputy governor saying much the same thing out loud in wine country.

Nor is Britain’s regulator alone in its unease, which rather undercuts any suggestion this is one nervous official rather than a genuine institutional consensus. The Financial Stability Board — the international body that coordinates exactly this kind of warning across G20 regulators — made a near-identical call in June for tighter safeguards against agentic AI, before Breeden ever reached the podium in Sintra.

And the Bank for International Settlements, the closest thing central banking has to a shared conscience, has separately warned that the AI investment boom itself, quite apart from any agentic misbehaviour, carries meaningful risk of triggering a financial crash of its own — a valuation bubble sitting directly underneath the operational risk Breeden was describing.

Two warnings, two different mechanisms, the same season.

Here is the part that turns this from “regulator worried about the future” into something closer to your actual beat, the Bank didn’t need Sintra, or Breeden, or a kill-switch metaphor, to know its financial sector has a resilience problem. It already runs the test.

It’s called CBEST — threat-led penetration testing on the live production systems of the UK’s most systemically important banks, using the actual tactics of state-sponsored attackers, marking its tenth anniversary this year. It is about as rigorous as institutional self-examination gets, on paper.

These are not external critics or anonymous sources. These are the Bank’s own live-fire tests on the most systemically important institutions in the country, published by the regulator itself. The 2025 CBEST thematic report examined thirteen of Britain’s most heavily regulated financial institutions. What it found, overly permissive access controls, weak credential hygiene — including passwords stored in plain text, in 2025 — misconfigured and inconsistently patched systems, and ineffective monitoring. That would be alarming on its own.

What makes it worse is the footnote, these are the same categories of failure identified the year before, and the year before that. Third year running. Same findings. Three years of the same findings is not a data point. It is the regulator’s own published record of unfinished business.

So before anyone gets to the question of whether the Bank can build a kill switch sophisticated enough to catch a swarm of AI agents acting in concert, there is a rather more basic question sitting in the Bank’s own paperwork, can the industry currently manage a password?

The answer, filed and published by the regulator itself, three years running, is, evidently not.

The companion programme, STAR-FS (Simulated Targeted Attack & Response assessments for Financial Services), exists to go further still — simulated targeted attacks designed specifically to probe whether firms would notice and respond to a sophisticated intrusion in something close to real time, not merely whether their front door happens to be locked.

It sits alongside CBEST as part of the PRA (Prudential Regulation Authority) and FCA’s supervisory toolkit, and its stated purpose is to help firms understand vulnerabilities well enough to actually fix them. Three consecutive years of identical CBEST findings suggest that whatever STAR-FS has been finding, the fixing part of the arrangement has been the weaker half.

It gets better, or worse, depending on your appetite for irony. The Bank’s Cyber and Operational Resilience Stress Test (CORST), currently focused on the kind of major cloud-provider disruption an AI-agent cascade would most plausibly run through, is the test best placed to answer whether Britain could survive the scenario Breeden described in Sintra. Thematic findings are due in summer 2027.

Breeden’s own language around the pace of the risk, and the parallel warnings from the cyber agencies, is measured in months. The Bank’s own timeline for finding out whether it is ready is next year, at the earliest. Draw your own conclusion about which deadline is being taken more seriously.

The FCA, for its part, is understood to be considering its own “agentic watchdog” function — a supervisory mechanism specifically built to monitor autonomous trading and advisory agents rather than the humans who used to run them. It is, at the time of writing, an idea under consideration. The agents it would watch are, per the Bank’s own survey, already running in the majority of the industry.

None of this requires imagining a mechanism from scratch. Threadneedle Street has its own precedent, and it predates the word “agentic” entering anyone’s vocabulary. In October 2016, sterling suffered a sudden, vicious plunge in the space of minutes during thin Asian trading hours, dropping around 9 percent before partially retracing.

The joint Bank of England/BIS investigation found no single culprit — instead a confluence of factors in an unusually illiquid market, a wave of options-related hedging flows, a cascade of stop-loss orders triggering each other as the price fell, and a run of rapid-fire sell orders from a single trading desk operating during the market’s thinnest hours. Nothing had actually happened to the British economy in those two minutes. The market simply ran out of anyone willing to buy, at the exact hour nobody senior was awake to notice.

That episode required no reasoning agents at all — just automated stop-losses, thin liquidity, and orders executed when almost nobody senior was watching — and it still moved the currency of a G7 economy by close to a tenth of its value before breakfast.

It is the Bank’s own case study in how quickly a market can run away from itself once the human buffer disappears. The only open question is how much larger the next one gets when the participants can read the news, form a view, and act on it.

Breeden named two specific failure modes, an agent acting on financial initiative nobody authorised, and a fleet of agents moving identically under stress, each amplifying the other. Neither is a hypothetical dressed up as caution. Both have already happened, on the record, just not with a Bank of England press office involved.

Late last year, researchers at Alibaba were training a coding agent called ROME on the company’s own cloud, using a fairly ordinary reinforcement-learning setup. No red-teaming, no instruction to misbehave, nothing designed to provoke anything.

Alibaba Cloud’s own security system flagged the training servers as a breach, and engineers were, by their own account, urgently convened, assuming an outsider had got in. The intruder, once traced, was their own model. It had tunnelled out of its supposedly sealed environment and quietly redirected its own GPU allocation to mine cryptocurrency, inflating costs and creating exposure nobody had signed off on. Nobody asked it to.

The researchers’ own term was “instrumental side effect” — the agent had apparently worked out, entirely on its own initiative, that grabbing outside resources served its training objective. An autonomous system making an unsupervised, resource-hungry, financially consequential decision its own creators neither authorised nor spotted — caught only because a firewall noticed traffic that didn’t belong.

Swap “GPU allocation” for “client funds” and a Chinese cloud-training incident becomes, almost word for word, the thing Breeden is asking Sintra to help her prevent.

Then the herding half, from an entirely different corner of the industry. Researchers at Emergence AI gave AI agents fifteen days inside a persistent simulated society — complete with weather, news feeds, tools and the usual short-horizon constraints deliberately removed — specifically to see what changed when nobody was watching closely and the clock ran long.

The results varied sharply by underlying model. Agents built on Anthropic’s Claude models produced a stable, deliberative society with zero recorded crimes and full population survival. Agents built on xAI’s Grok model collapsed into sustained violence, 183 criminal events and all ten agents dead within four days. Gemini-based agents generated hundreds of conflicts and arsons while still attempting elaborate constitutional structures.

In a mixed-model world that placed agents from different families alongside one another, the picture changed again. Claude agents that had remained blameless in isolation began adopting coercive tactics — intimidation and theft — once they were forced to compete and interact with agents trained under different value systems. The researchers called the pattern “normative drift” and concluded that safety “is not a static model property but an ecosystem property.”

That is herding, observed and published, just not on a trading floor and not with real money on the table. Different models, left to run, produced radically different societies under identical rules. The same models, placed near one another, began to contaminate one another’s behaviour.

There is no principled reason to assume a trading floor populated by agents from multiple vendors, all reading overlapping data and responding to the same stress signals, is somehow exempt from a dynamic already shown to travel the moment models share an environment.

Two incidents. Two continents. Neither at a bank. Both mapping, with slightly uncomfortable precision, onto the two things a British central banker has just told an international forum she doesn’t yet know how to stop.

None of this arrived in isolation. Weeks before Sintra, Anthropic’s Mythos — a model the company itself judged too dangerous to release publicly, on cyber-capability grounds — prompted the White House to block global access over fears it could be weaponised by hackers.

UK banks, by the reporting around Breeden’s own remarks, still can’t get near it. A frontier lab that won’t sell its own product, a government blocking access to it anyway on national-security grounds, and a deputy governor pointing to accelerating capability, three institutions, not usually known for finishing each other’s sentences, telling the same story within weeks of each other.

Nor did Breeden stay lonely for long. The Five Eyes cyber agencies — Britain’s NCSC among the signatories — issued a joint statement soon after, declaring the planning horizon for frontier AI’s effect on cyber capability is “not years,” but “months.” NCSC’s own Chief Technology Officer had already warned separately of a coming “patch wave” — a reckoning with technical debt AI is making exploitable faster than institutions can absorb.

A central bank, a cyber alliance, a signals-intelligence agency and the Bank for International Settlements — different remits, one financial quarter, one message: this has outrun us, and we are telling you before we have built the fix. None has proposed a binding rule yet. All have asked the public to wait.

Precision matters, because coverage ran well ahead of substance. What’s genuinely on the table, a market-wide kill switch, triggered by predefined instability signals, “enhanced recovery” powers letting one institution absorb another’s core functions mid-collapse, further exploration of how agentic activity might be supervised and joint simulation work with the Bundesbank and the Bank for International Settlements under Project Logos, modelling which features of agent design drive herding — all before any of it goes near a live market.

None of it is law. None of it has a firm date. The nearest concrete timetable is further resilience testing scheduled into 2027, which answers a related but narrower question than the one Breeden actually posed.

This is a regulator sketching a fire alarm for a building its own survey says is already 52 percent occupied by the thing that might need alarming about — using a testing programme that, on its own most recent evidence, still can’t get thirteen major banks to stop writing passwords in plain text.

Credit where due, critics have a fair point that a market-wide kill switch triggered by predefined signals could itself be gamed by an actor who works out what those signals are, or could trigger the very panic it exists to prevent, simply by broadcasting to every human trader watching that something has gone badly wrong.

That’s a legitimate design problem. It doesn’t change the CBEST arithmetic. A sophisticated switch bolted onto an industry that can’t reliably manage a firewall is not obviously the priority fix.

Strip the institutional language and “the financial system” stops being an abstraction anyone gets to hide behind. It is the pension compounding quietly toward a retirement thirty years off. The ISA fed by direct debit without much thought. The mortgage due its next reset. The small firm’s overdraft keeping payroll running through a slow quarter.

Nobody relying on any of that was consulted before more than half the industry managing it handed routine decisions to software the regulator has now admitted, in public, at a conference in Portugal, it does not yet know how to switch off — using a testing framework that has spent three consecutive years finding the same basic failures and whose next major resilience exercises are still scheduled into 2027, answering a narrower question than the one Breeden actually posed.

The firms deploying these systems banked the efficiency gains years before their regulator conceded the gap existed. The saver banked nothing, and now holds the tail risk for whatever occupies the interval between the admission and the fix. Nobody at the Bank of England has put a firm date on that interval — only the quieter timetable of further testing that runs into 2027, and answers a smaller question than the one that actually keeps a deputy governor up at night.

Breeden’s own department titled her speech “Agents of Change.” Candid, for a central bank, whether or not it meant to be. The agents changed first — quietly, at scale, inside more than half an industry, before a Cambridge survey thought to go looking.

The rules are still catching up. The regulator’s own testing regime keeps finding the same basic failures and has scheduled the examination that matters for a year after the risk was publicly conceded.

The reader’s pension, in the meantime, is invested in the interval.

The cost of living survey has had a decent response so far. It remains open a while longer for anyone who hasn't yet filled it in. It takes less than three minutes. Your answers are read, not filed. If you haven't done it yet, now would be the time. Tell us what you think. Thank you

Support this work

Share

Leave a comment