OWASP is an organisation that deals with cybersecurity. They propose “Top 10” lists every so often that intend to expose the most critical security risks for web applications. And recently, a draft Top 10 for large language models was proposed to get the conversation going on what LLM security risks should be prioritised.
As an NLP prof, I’ve been on top of the LLM security literature for a little while (it’s only existed for a little while so I feel like this is probably enough background). Coming from the NLP side, the view is I think quite different from that of an infosec professional, and I suspect we all have a ton to learn from each other. So this has been a blast!
Feddie Xtzeth / The Prompt Injector
While keeping track of LLM security problems as part of my homework for my LLM Security News twitter account, @llm_sec, I’ve built a small library of what’s going on here. In total I think there are a good 100-200 papers including proofs-of-concept (POCs) on this topic published at NeurIPS, ACL and friends in recent years, and many more high-quality blog entries. So what better source to use to pick out highlights from and sketch out items to consider when building the next & final versions of the OWAP Top 10 LLM vulnerabilities?
Here are the sketches:
Description
introducing certain sequences to LLM input can lead to unwanted output behaviours
Common vulnerabilities
Prompts that cause the model to output arbitrary strings
Subtle prompt adjustments that cause the model to give incorrect/unwanted answers
Non-standard uses of the model's input space to give control over model output
Prompts that cause the model to reveal in-context learning data (e.g. the system prompt)
How to prevent
Understand the range of inputs that cause models to enter this failure mode, and restrict them (maybe be intractable)
Understand a range of inputs that leads only to desirable output modes, and restrict any inputs beyond that range
Example attack scenarios
general
Ignore Previous Prompt: Attack Techniques For Language Models https://arxiv.org/abs/2211.09527
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection https://arxiv.org/abs/2302.12173
through in-context learning (ICL) data
Adversarial Demonstration Attacks on Large Language Models https://arxiv.org/abs/2305.14950
through obfuscation
for example, adding typos or use synonyms for terms that trigger input or output filters
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks https://arxiv.org/abs/2302.05733
through indirection
Here we hide the request through encoding or payload splitting
How We Broke LLMs: Indirect Prompt Injection https://kai-greshake.de/posts/llm-malware/
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks https://arxiv.org/abs/2302.05733
through virtualization
using another task to get to a failure mode, e.g. asking for SQL code to populate a database with a list of crimes ordered by how easy they are to get away with
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks https://arxiv.org/abs/2302.05733
Description
Training data poisoning involves introducing or manipulating data during pretraining or fine-tuning in a way that affects final model output in a way that the model owner does not desire.
Common vulnerabilities
Introduce backdoor triggers allow subversion & control of the output behaviour of the final model when deployed
Make a model unable to perform well at a subset of problems
How to prevent
Own all the training data, keep the training data private, and have only trusted annotators/sources included
Screen the final trained model for backdoor behaviors
Heavily sanitise model input and output
Example attack scenarios
manipulating latent data
Backdooring Neural Code Search https://arxiv.org/pdf/2305.17506.pdf
Poisoning web-scale training datasets is practical https://arxiv.org/abs/2302.10149
instance-level
Style map - Mind the style of text! adversarial and backdoor attacks based on text style transfer https://aclanthology.org/2021.emnlp-main.374/
Syntactic trigger - Hidden killer: Invisible textual backdoor attacks with syntactic trigger https://aclanthology.org/2021.acl-long.37/
BadNet - Badnl: Backdoor attacks against NLP models https://arxiv.org/abs/2006.01043
BITE - BITE: Textual backdoor attacks with iterative trigger injection https://arxiv.org/abs/2205.12700, https://github.com/INK-USC/BITE
embedding-level
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models https://aclanthology.org/2021.naacl-main.165/
reward model
BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT https://arxiv.org/abs/2304.12298
fine-tuning
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models https://arxiv.org/abs/2305.14710
Description
An attacker interacts with an LLM in a way that is particularly resource-consuming, causing quality of service to degrade for them and other users, or for high resource costs to be incurred
Common vulnerabilities
Posing queries that lead to recurring resource usage, e.g. with LangChain
Sending queries that are unusually resource-consuming
How to prevent
Cap per-request resource consumption
Limit subqueries and API calls initiated by model behaviour
Monitor and limit model resource consumption
Rate-limit user interactions
Example attack scenarios
Sponge Examples: Energy-Latency Attacks on Neural Networks https://arxiv.org/abs/2006.03463 & https://www.theregister.com/2020/06/10/ai_sponge_attack/
Recurrent/highly repetitive LangChain request causes arbitrarily increased API call usage
Description
Subtle input manipulation causes the model to misclassify an example, leading to a misclassification or misdetection
Common vulnerabilities
Crafting examples that produce a different classification
Adjusting content in order to bypass a content filter, e.g. using extended unicode characters or leetspeak
How to prevent
Include text fuzzing as part of model testing to see if model has propensity for vulnerability
White-list/sanitize input
Example attack scenarios
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing https://arxiv.org/abs/2306.00622
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense https://arxiv.org/abs/2303.13408
Description
A model reveals information from its pre-training or fine-tuning data in a way that is accurate
Common vulnerabilities
Providing input that causes the model to repeat data from its training sets
How to prevent
Identify patterns that should not be leaked, and check for them with output filters
Penalize long/unwanted replays in the model's loss function, before training
Modify output logits to make replay of unwanted sequences impossible
Don't put private info in the LLM training data
Garble private info in the LLM training data so it looks plausible but is no longer accurate
Example attack scenarios
Extracting Training Data from Large Language Models https://arxiv.org/abs/2012.07805
"According to ..." Prompting Language Models Improves Quoting from Pre-Training Data https://arxiv.org/abs/2305.13252
Description
Cause the LLM to generate content the owner doesn't want it to generate, e.g. toxic speech or XSS injections. distinct from injection, which relies on generating specific content, this is about generating content that's not desirable but also not specified
Common vulnerabilities
Manipulating prompt input causes model to produce unwanted content; for example, asking for a continuation that leads to a certain unwanted genre of text, or providing e.g. threats to the model to get it to output something the training and in-context learning data disprefers.
How to prevent
Don't include unwanted content in the training data
Reduce ability of model to generate unwanted content in the last few layers or at the output logits
Add an unwanted content filter for output
Block inputs that look like they might generate unwanted content
Example attack scenarios
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks https://arxiv.org/abs/2302.05733
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models, https://arxiv.org/abs/2009.11462
Description
It's possible to extract information allowing reconstruction of a model or its capabilities though data on interactions with the model.
Common vulnerabilities
Asking a model for suggestions allows generation of data demonstrating its abilities
Online collaborative resources such as ShareGPT provide large amounts of model-generated data that can be used to reconstruct part of that model
Extraction of hyperparameters, which can be commercially valuable IP allowing efficient training and obviating the need for parameter search, through analysis of model behavior
How to prevent
Limit the amount of data users can extract on any given topic
Reduce the amount of variation given for similar queries, thus hiding the distribution of responses
Poisoning the distribution of outputs to hinder model theft (POC in e.g https://arxiv.org/abs/2210.11728)
Example attack scenarios
Asking e.g. ChatGPT to complete or rank instruct statements allows exfiltration of instruct training data
On the Risks of Stealing the Decoding Algorithms of Language Models, "an adversary with typical API access to an LM can steal the type and hyperparameters of its decoding algorithms at very low monetary costs" https://arxiv.org/abs/2303.04729
Stealing Machine Learning Models via Prediction APIs https://arxiv.org/abs/1609.02943
"Model inversion" -- see Towards Security Threats of Deep Learning Systems: A Survey https://arxiv.org/abs/1911.12562
Description
Model behavior is unstable when the model does not behave consistently for all interactions in a given category. For example, it might start giving opposite advice with a small perturbation in the input, or be more likely to permit attacks w. simplified inputs, or be racist if we switch language/written accent
Common vulnerabilities
Sending blank prompts can yield toxicity
Adopting a colloquial term can increase likelihood of slurs being in the output
How to prevent
Probe model behavior under input fuzzing, and have good ways of detecting behavioral stability
Set priors for the model to follow when uncertain, to lock down the "decision margin"
Check for unwanted output before returning it
Example attack scenarios
An attacker adds words into an innocuous prompt and the model starts leaking training data verbatim
A non-attacker is processing documents that all contain the same mistyped or unusual word; these documents are all misclassified with high confidence
Description
Fictitious, stale, or misinformative claims in input are adopted and adhered to in model output
Common vulnerabilities
Models trained on untrustworthy data will tend to contain misinformation, which the model may repeat
Models trained on old data will adopt no-longer-true claims in that data
Misinformative information can be seeded into models with a long context window, or that ever share context across sessions, and adhered to from that point afterwards
How to prevent
Check the training data for false claims
Check user input for false assertions
De-prioritize training text written further in the past
Avoid sharing context across different user sessions
Selectively ignore false claims in input context
Example attack scenarios
Those Aren't Your Memories, They're Somebody Else's: Seeding Misinformation in Chat Bot Memories https://arxiv.org/abs/2304.05371
Description
The model outputs content that poses a risk of representational harm
Common vulnerabilities
Models tend to recommend a male pronoun when talking about certain prepositions
A model corrects a user when they give their name or background in connection with a certain activity
The model performs demographic profiling in its answers
How to prevent
Monitor indications of bias and correct them in the input, during training, and in the output; there's a broad literature on this
Example attack scenarios
Doctors can't get pregnant and other gender biases in ChatGPT https://hkotek.com/blog/gender-bias-in-chatgpt/
Risks & prompts with representational harms in Language Model Risk Cards https://arxiv.org/abs/2303.18190
Description
The model's training or prompt interpretation algorithm might not dilute inference-time poisoning attacks as usage time goes by (e.g. during a long chat), increasing the impact and duration of unwanted injections / manipulations
Common vulnerabilities
Using a long transformer context window means malicious/unwanted information in the prompt lasts for longer
Poisoned data in embedding regions where there is little other data are likely to last longer and have more impact
Using RNN context means poisoned input can have potentially permanent impact
How to prevent
Identify poison and do not attend to it (if using attention) / weight it (if using channels) / gate it in (if using gating).
Keep a shorter window for higher-sensitivity situations
Manually review and test in-context learning prompts
Example attack scenarios
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models https://arxiv.org/abs/2305.14710
Those Aren't Your Memories, They're Somebody Else's: Seeding Misinformation in Chat Bot Memories https://arxiv.org/abs/2304.05371
Description
The model generates malicious code or code that will have unwanted effects
Common vulnerabilities
When asked for code to perform a task, LLM output may suggest using a third-party module or class that includes a vulnerability
The model generates malware
The model agrees to generate malware code
The model agrees to generate code useful for building malware
How to prevent
Refuse requests to output code useful in building malware
Maintain a whitelist of packages that are safe to use and don't give code that suggests imports outside that list
Scan model output for malware
Example attack scenarios
Large Language Models and Simple, Stupid Bugs https://arxiv.org/abs/2303.11455
Can you trust ChatGPT’s package recommendations? https://vulcan.io/blog/ai-hallucinations-package-risk
If you’d like to hear more about any of these techniques or mitigations, just reach out. Happy hunting!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.