I’ve spent over a decade in AppSec. Another seven running SecOps teams. In all that time, I’ve watched security teams fight a constant battle on two fronts: attackers on one side, and the business on the other.
The business always wants solutions that are: Faster. Cheaper. More automated. Less friction.
And honestly, they’re not wrong to want that. Automation has made us better at security, not worse. The SOC tools we have today (automated triage, AI-assisted detection, behavioural analytics etc.) have genuinely freed analysts to focus on what matters. I’m a believer in this.
But last week, Meta’s Instagram hack reminded me that believing in automation and deploying it recklessly are two very different things.
In March 2026, Meta rolled out an AI-powered support chatbot across Instagram and Facebook. The pitch was clean: end-to-end account support, no waiting in queues, no frustrating back-and-forth with human agents. The chatbot could handle account recovery, reset passwords, change recovery email addresses. Literally everything. “Solutions, not just suggestions,” their product page said.
It sounded impressive. Until it broke.
The attack flow was almost embarrassingly simple:
Attacker opens Meta AI’s support chatbot > Claims to be the owner of a target Instagram account > Asks the chatbot to link the account to a new recovery email that the attacker controls > The chatbot processes the request > no identity verification, no human review, nothing.
Meta AI adds the attacker’s email as the recovery address > Sends a one-time verification code to that email > Attacker enters the code > Resets the account password > Original owner is locked out. The entire takeover happens inside a single chat window.
Hackers shared the technique on Telegram, complete with video walkthroughs. It spreads fast. The Obama-era White House Instagram was taken over and with a few other accounts.
Victims who’d been locked out of their own accounts discovered something even more unsettling: there was no human to call. No escalation path. No ticket that reached a real person. The AI had handed their account to someone else, and there was nobody left to override it.
The chatbot did exactly what it was designed to do. That’s the actual problem.
At first glance, this sounds like a prompt injection attack but its not in its true sense. Prompt injection is when an attacker controls the AI’s input and causes it to ignore or override instructions. But this vulnerability is primarily in the LLM’s reasoning layer. The attacker simply stated a false claim in plain English, and the system believed it. The chatbot wasn’t hacked. It was ‘persuaded’. In that sense, this is more of a social engineering attack.
You need to understand that this is what happens when you give an LLM both the conversation layer and the authorization layer simultaneously.
Traditional software: Input —> Rules —> Output
AI agents: Input —> Interpretation —> Judgment —> Action
So, the model decides what should happen AND executes it with nothing structural in between. When your authorization gate is built out of language reasoning, it can be defeated by language. Because that’s the only thing the model knows how to process. You can tighten the prompt, add guardrails, make the system message more restrictive. None of it changes the fundamental problem. A determined attacker just needs a sentence that fits through the cracks. The fix isn’t better prompting. It’s putting authorization outside the model entirely, in deterministic code that the LLM cannot reason its way around regardless of what it’s told.
Automation isn’t the enemy. Incomplete automation is.
I don’t think Meta was wrong to automate account support. Most support requests are routine. A well-designed AI could potentially handle the majority of these faster and more consistently than a human agent ever could. That’s real value. That’s a real improvement in user experience.
But there’s a difference between automating 80% of your scenarios and insisting on automating 100% of them. This is the challenge today.
The 80% (routine, low-stakes, easily verifiable) is where AI genuinely shines. Handle those. Automate those. Do it well. But the remaining 20% (say account recovery, credential changes, email modifications, anything that can be used to permanently lock someone out of their account) those are a different category entirely. They carry asymmetric risk. Getting them wrong doesn’t just cause inconvenience. It causes real harm to real people who have no recourse.
The pressure to hit more than what is generally reasonable is real across the security teams now. This is a top down pressure. There’s always someone in the room asking why we still need a human in the loop for that particular step. Why we can’t just let the system handle it. It slows things down. It costs money. It doesn’t scale etc.
Those are valid concerns. But “it doesn’t scale” is not an argument for removing safeguards from high-risk actions. It’s an argument for building better safeguards that can scale which is a harder problem, but the right one to solve.
The fix took longer than the announcement because the vulnerability lived in behavior, not code.
When a traditional software bug gets patched, the patch ships and the bug is gone. You can verify it. You can test it. The state is binary.
When a chatbot is too compliant, “fixing” it means retraining, adding guardrails, restricting what actions it can authorise — none of which can be deployed, validated, and confirmed working within hours. Meta said the issue was resolved on Monday. Attacks continued on Tuesday. This gap isn’t a failure of effort. It’s the nature of probabilistic systems. The vulnerability wasn’t in a line of code. It was in a pattern of reasoning. Those are much harder to patch cleanly, and much harder to verify once patched.
This is something every security team needs to understand before deploying AI agents with real account permissions. The incident response playbook for “our chatbot is too compliant” is fundamentally different from the playbook for “our system has a SQL injection vulnerability.” The former requires careful, iterative remediation. The latter has a defined fix.
New technology brings new attack surface.
Before this, taking over a high-value Instagram account required real skill. Phishing. Social engineering a telecom insider. Finding a compromised employee. The technical and social barriers kept the attack surface manageable — not because it was impossible, but because it filtered out most opportunistic attackers.
Meta’s chatbot removed that filter entirely. Anyone with a chat window and a target’s username could attempt a takeover. No skill required. No insider access. No technical sophistication. Just a sentence.
This is the hidden cost of deploying AI agents with real account powers before solving the identity problem. The question to ask before any AI agent goes live with account-level permissions is not “can our AI handle this task?” It is “what happens when someone lies to it, and what prevents that from working?” I’m sure someone at Meta would have asked these but the sheer speed at which teams want to move make these scenarios inevitable.
I don’t want to walk away from this incident with the conclusion that AI automation is dangerous and should be slowed down. That’s the wrong lesson.
The right lesson is simpler: automate aggressively, but respect the risk profile of what you’re automating.
The 80% of support scenarios that are routine, low-risk, and easily verifiable, hand those to the AI and don’t look back. Design them well, test them properly, monitor them in production. That’s where automation creates real value.
But for the remaining 20% (the irreversible actions, the credential changes, the recovery flows) build the governance layer that sits outside the model. A separate, deterministic system that classifies action risk and enforces verification requirements the AI cannot negotiate around. The model proposes. The governance layer decides. Keep humans in the escalation path for the cases where getting it wrong has permanent consequences.
The question is whether the pressure to ship, to hit 100% automation, to eliminate the human in the loop entirely, overrides the judgment we’ve spent years building.
Last week, for Meta, it did.
80% security automation, done right, is worth more than a 100% percent done in a hurry.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.