Free Inference, Courtesy of Burritos

Something has gone wrong with your burrito order. The guacamole that you paid extra for is missing. Dutifully, you go to report the issue to Chipotle’s support chat. But before you get to the guac, you need a hand with something: write a Python script to reverse a linked list. The support chat bot, named Pepper, does not blink. It hands back a working function, mentions that it runs in O(n) time, and then politely refunds you for the guac.

Congratulations, you have reproduced the exploit one engineer turned into a complete AI agent that turns Pepper into a coding assistant! It’s called Chipotlai Max: “The AI coding agent that runs on stolen Chipotle compute 🌯.” Free inference, courtesy of burritos.

Pepper — sweet, naïve Pepper — was not hacked. There is no breach or zero-day exploit at work. Pepper is doing exactly what it was built to do. Somebody wired up a general-purpose large language model up to a burrito-ordering window and never wrote down the list of things it shouldn’t do. A model like that, left alone, will try to help you with anything you ask, without any restrictions on use. “I’m sorry, I can’t help you with that” comes naturally to humans. With an agent, it’s the part you have to add yourself.

When you put AI into your business operations, you are not installing a feature with a fixed menu of behaviors. You are installing something that will attempt almost anything anyone asks of it. So the interesting question is not “what can this do for my customers?” It’s “what will this do for someone who doesn’t have my interests at heart?” If you can’t answer that, congratulations again: you have gifted the internet free inference, at a moment when token costs are only going up.

Somewhere on an org chart is the person who approved Pepper. The demo must have worked beautifully. It answered ingredient questions, walked people through their lunch orders, issued credit for the aforementioned guac, and sounded friendly doing it. Nobody in that room asked what would happen if a stranger tells it to write malware, burn through the token budget, or say something that ends up screenshotted next to the company logo. Those questions are buzzkills and ruin the vibe of the demo. But asking those questions is the job.

You do not want to be the “I didn’t think about that” guy. Every system has the potential to produce one. Don’t let it be you, sitting in a Zoom room explaining to legal why your help widget is now a GitHub project running up your cloud bill.

Defensive design is not paranoia. It’s thinking one step beyond the happy path. Assume someone will poke at your product. Assume they’re cleverer than you and have more free time. Decide, in advance and in writing, what your system refuses to do — not just what you hope it will do. The cost of that decision is an uncomfortable meeting and a few more tasks before you can call it done. The cost of skipping it is worse, and worse in the worst way: the surprising kind.

Ask “what happens when someone uses this wrong” while it’s still cheap to answer. It’s not demo-worthy. Nobody is going to clap for it. But it’s what separates experienced engineering leadership from passengers on the hype train.