You Can Delete the Chat. Not the Learning

After I published “Spilling Secrets” back in March, I went to practice what I preached. Opened my own AI settings, found the “help improve the model” toggle, flipped it off. Felt responsible for about a day.

Then I read the next line of the policy. Opting out only governs what happens next. It says nothing about what I’d already sent.

Here’s the part that matters.

The delete button feels like a guarantee. It’s actually bounded to whatever’s still tied to your name.

Anthropic, in its own retention terms, spells it out: turn off training and future chats won’t be used but data already folded into a training run in progress, or into a model that’s already been trained, stays where it is.

If you leave training on, that retention runs up to five years.

OpenAI will delete your chats within 30 days, unless they’ve already been de-identified and disassociated from you, in which case the 30 days doesn’t apply.

Google’s Gemini pulls a sample of conversations for human review. Once a sample is selected, it’s disconnected from your account, and Google says it can’t be deleted through your history and they keep it for three years. The default is 18 months.

Three companies, three different toggles. Same quiet admission underneath.

Once your data is separated from you — trained into a model, de-identified, or pulled into a review sample — the delete button stops reaching it. Deletion isn’t the default. It’s a feature you assumed you had. Permanence is what you actually get.

The short take:

You can delete the chat. You can’t delete the learning.

This is where AI stops being a productivity question and becomes a security one. The old security rule was simple: you protect what you can delete. But you can’t secure what you can’t delete. And once it’s in the model, you can’t delete it.

If that sounds like a future risk, it isn’t.

Just this year, a security researcher found the message database of Chat & Ask AI — one of the most popular AI chat apps on the App Store and Google Play, with more than 50 million users — sitting open on the internet. Roughly 300 million messages from over 25 million users, as 404 Media reported. People discussing illegal activity. People asking for help ending their lives.

Here’s the part that should bother you.

The app is a wrapper — it plugs into ChatGPT, Claude, and Gemini and lets users pick their model.

You could read OpenAI’s retention policy, or Anthropic’s, and feel covered. It didn’t matter. The middleman had left its database publicly readable; a misconfiguration so common the researcher built a scanner for it and found the same hole in 103 of the 200 iOS apps he checked.

The developer fixed it within hours of being told. Credit for that. But the fix only closes the door going forward. Every message already in that database was exposed for as long as it sat there. The send was the irreversible act. Everything after it was cleanup.

I’d like to tell you I only watch other people do this, but that would be a fib.

I was building an email app, using Claude Code to move faster, and caught data leaking out through the tool while I worked. My own project. My own mistake. The uncomfortable part wasn’t that it happened; it’s that I wouldn’t have noticed if I hadn’t been looking.

So I changed how I work, and I’ll tell you what I do, not what you should do.

• I keep a manifest. Every AI model and service my agents touch, with the date I last checked its privacy terms. When a new tool shows up, it gets screened before it gets adopted — not after.

• I avoid AI where plain code does the job. If I need to sort messages by a known subject line or a from-address, that’s a rule, not a model. No inference means nothing to send.

• I use local models if I can. When sensitive data genuinely needs a model, I run one locally with Ollama, so nothing leaves the machine. And when local won’t do, I sanitize first — strip the data down until it can’t be linked back to a client or an account.

A major mistake I see is allowing your coding agent to use the LLM when there are better, non-AI tools for the job. For my email project, I was able to use a number of email classifying tools that once a local LLM trained the rules, AI was no longer needed.

An even bigger risk is your vendors. If you outsource AI work or integrate with third parties, you may not know how they are handling privacy on the data they submit on your behalf.

So check your vendor’s policy on privacy when using AI. You may not want anyone learning that you actually like pineapple on your pizza.

TLDR; Once it’s in the system, you may not get it back out.

So screen the AI projects you run, and the ones being built for you, for one thing: what happens to the confidential content they touch.

I’m Jeff Huckaby, founder of rackAID. I help established businesses make sound technology decisions — connecting technical activity to business outcomes.