Simon Willison coined the term “the lethal trifecta” to characterize three capabilities of AI systems that together make the resulting AI system unsafe:
Private data + access to untrusted content + external communication = risk of data exfiltration
There are many ways in which an AI system could be “unsafe”. In this model we’re specifically thinking of data exfiltration attacks:
- Private data is something that an attacker would want to exfiltrate.
- Untrusted content is what the attacker would use to deliver subversive instructions (prompt injection) to the agent.
- Once the attacker has control over the agent, they would make it send the private data to the attacker over the external communication channel.
As simonw@ noted in another post :
The whole point of the lethal trifecta framing is that the only way to reliably prevent that class of attacks is to cut off one of the three legs!
What happens when we remove one of the legs? Let’s consider the three scenarios.
The sandbox
In the first model, we want to remove the ability to communicate with the external world. That is, the system needs to run in an isolated environment. Such environments are often called “sandboxes”, so let’s call this category “The sandbox”.
At the same time, the agent does have access to untrusted content. The untrusted content could be a static data set, e.g. all the incoming emails that a company received this month. Or it could be an ongoing data source, e.g. a real-time stream of all the incoming email. The system can passively receive incoming untrusted data points from the outside world.
What we cannot allow is the system actively making requests for that external information, as any form of external requests creates a risk of data exfiltration. In the simplest case, the agent can be tricked into requesting a special URL that encodes stolen private data into query parameters which are then logged at the attacker-controlled external server. In an advanced scenario, timing of the request can be used as a low-bandwidth data exfiltration channel.
An example of the sandboxed system would be a research agent comparing internal research data with a body of external publications on the same subject. This system can be fully cut off from the outside world. Upon completion, it presents a research report for internal consumption.
The lab
In the second model, we want to ensure the agent doesn’t see any untrusted content. That is, there should be no way for the attacker to slip in subversive instructions to the agent. I call it “the lab” to evoke a strictly controlled environment of bio-chemistry laboratories.
At the same time, we do allow the agent access to valuable private data and we give it the ability to communicate with the external world.
The problem with this approach comes from the premise of somehow allowing external communication, but without exposing the agent to untrusted content. This can mean one of two things:
Option 1: we allow unconstrained communication with an external party, but we trust that party to only send safe content to the agent. This case isn’t very interesting, because in this model the external party becomes indistinguishable (trust-wise) from ourselves; the communication is not meaningfully “external”.
Option 2: we constrain the communication protocol so that it’s not possible to use it to slip in subversive instructions.
As an example of how Option 2 could look, consider a trading agent that has access to internal financial models and data and can also query current financial parameters from the outside world.
The more powerful/expressive the API used to retrieve the information from the outside world, the higher risk of attack. For our example of the trading agent, we have a few options for the API design, from lowest to highest risk:
- An API for retrieving stock prices. The agent can request a company NASDAQ ticker, the API returns a number. This is probably safe.
- An API for web search returning a summary of search results. E.g., the agent could ask for “financial markets news”. The external service reads the financial news and returns a summarized response. The summarization step makes it harder, but not impossible, for the attacker to sneak in subversive instructions to our agent. And if the attacker compromises the web search service, they’re only one step away from hijacking our agent.
- An API for web search returning full-text results from matching websites. In this case our agent is directly exposed to untrusted content, and we’re no longer in the “lab” model.
The honeypot
In the last model, we give up on protecting our sensitive data and make sure the agent doesn’t have any of it in the first place. The agent will have access to untrusted content and external communication channels, making it vulnerable to prompt injection.
I call it “the honeypot”, because I struggle to see many practical scenarios in which an institution or a private individual would be comfortable with their agents being hijacked by attackers.
The only use cases I can think of are research experiments: putting up agents with access to external communications specifically to test their ability to detect prompt injection and to collect data on incoming attack attempts, hence I’m calling this model “the honeypot”.
Conclusion
The lethal trifecta model doesn’t give a recipe for building a system that’s generally safe, it only helps to see some systems as evidently unsafe in the context of data exfiltration attacks. To stay clear of that specific risk, we need to remove one of the three elements.
If we remove external communication, the attacker won’t be able to retrieve our private data, but they can still hijack the agent in the sense of manipulating its work. (The sandbox.)
If we remove untrusted content, we eliminate the risk of prompt injection, but risk bringing it back when the external communication protocol is sufficiently expressive. (The lab.)
Finally, we can just give up on preventing the prompt injection hijacks and make the agent unabashedly vulnerable, while not giving it any private data so that there’s nothing that can be exfiltrated. (The honeypot.)
Each of those approaches is constraining, and that’s the price we must pay for avoiding the “lethal trifecta” attack scenario.