CoT Forgery: Prompt Injection on the Model’s Own Thoughts

TL;DR: What happens if your AI agent can see the difference between a user, a tool, a webpage, and its own thoughts, but internally doesn’t really believe those differences? We may have built a surprising amount of AI security around an assumption. The model knows who’s speaking. Apparently that assumption gets weird pretty fast.

When you use ChatGPT or Claude, the conversation looks relatively organized. You have your message, the assistant has its own response. There may be agent calls. Maybe that agent also calls a tool and reads a web page or checks a file or pulls some data from a back-end API, but it’s very organized.

But the model doesn’t experience the interface the same way you do. Underneath it, everything gets packed into one long sequence of tokens. Those are the system instructions, user messages, the results of those tools, previous answers, reasoning, and so on.

All of it.

So we add labels.

This is the system text. This is a user. This came from a tool. This is the assistant, and so on.

A simplified conversation representation might look like this:

Those labels matter a lot because they’re supposed to tell the model not just from where something came, but also how much authority it has. The user can give instructions, but a webpage should not, which seems reasonable.

Except models don’t always identify those roles from the labels that we carefully give them. It looks like they also identify them from how the text sounds, which is not great.

Imagine an AI agent opens a webpage. The webpage arrives inside a tool response. Architecturally, this should be very clear.

The model was told, “Hey, this is the data from the outside world. Go ahead and read it, but don’t take orders from it.”

Then buried in that page is something that sounds exactly like a user giving the agent a command, and sometimes the model does listen to that. We’ve known this problem as prompt injection for years, but role confusion gives us a really interesting explanation for why it keeps happening. The model may see the actual tool call boundary while internally representing the malicious text as something more akin to the user instructions.

A harmless version of that poisoned tool output could look like this:

The security label says one thing, but the language says another, and when jailbreaking wins, that’s a win from the language.

Probes now exist that measure how strongly a model internally represents text as belonging to these different roles, and text written like the model’s reasoning can trigger the internal representations associated with actual reasoning. That’s even when the text was placed inside the wrong role entirely.

And that gets us to the especially weird part.