Watermarked LLMs

A very fun, contentious recent moment has been Anthropic announcing watermarking of generated text. There's been a fair amount of misinformation and misunderstanding about the "watermarking". To be clear, there are no special characters, no hidden text, no fancy fonts- the watermarking comes from stylistic and semantic patterns artificially imposed on the text generation- and it's likely not as invisible as outraged blog posts would have you believe. Humans can detect textual watermarking (fingerprinting) fairly well. For instance, consider the following passage:

He wisely resolved to be particularly careful that no sign of admiration should now escape him, nothing that could elevate her with the hope of influencing his felicity; sensible that if such an idea had been suggested, his behaviour during the last day must have material weight in confirming or crushing it. Steady to his purpose, he scarcely spoke ten words to her through the whole of Saturday, and though they were at one time left by themselves for half an hour, he adhered most conscientiously to his book, and would not even look at her.

The content may tip you off to the fact that this is clearly Pride and Prejudice, but the self-correction, a vocabulary not of this period, and the sentence structure are all very much in the style of Jane Austen.

You've learned what Austen sounds like through reading her work and the period imitations of it (Downton Abbey, Pride and Prejudice and Zombies, etc.). To the point that when you read the passage you may have even elevated the voice of your inner narrator to a posh, English register. This "watermarking" is statistically evident as well; compare the top 10 words from Austen versus Cory Doctorow.

You do not need to be a literary scholar to see that Doctorow is very different from Austen.

LLM generated code is also full of tells in voice, pattern², and vocabulary¹; Anthropic's announcement, and their likely mechanism (assuming something along the lines of Kirchenbauer et al. 2023)- is a model for determining with a high degree of certainty that a given text was generated by Claude.

The watermarking, in essence, is a statistical game of madlibs. Let's play a quick game to drive the idea home. Fill in the following blanks:

You'll notice that I've only given you very Jane Austen-esque words to fill in. Hence, while the original sentence reads:

I don’t care who you are, there will always be times when hell is other people. Not because other people are horrible – quite the opposite! Other people are wonderful, but boy are they ever stubborn.

Your resultant quote likely sounds more like Austen than Doctorow. The watermarking method is essentially the same: LLMs (and in general the construction of sentences writ large) perform a grandiose game of madlibs. The watermarking in this case is restricting the vocabulary (as I did for you) in such a way that the "style" is evident, but the outputs are not harmed. In particular, the paper I've pointed to illustrates how using the previous word to generate the madlibs list of the next word can be used to this effect. Another example:

In this case from your first word, you're forced into to "style" that is easily detectable. While the "style" may not be in voice, as I've demonstrated here, the same math and idea applies.

The paper is even generous enough to catalogue an easy way of defeating the watermark! The "Emoji attack" works by alternating each word with a filler token (or rather and emoji), and then removing the filler, the watermark is easily defeated, but with a loss in quality.

It may just be recency bias, but even before reading this announcement I have noticed a stylistic decline and a loss of quality specifically in the Anthropic models³. For fun, I generated a list of 1000s of random numbers and asked it to rephrase a paragraph of some text with those numbers interspersed (hence providing a watermark break).

This triggered Anthropic's abuse detection.

API Error: <model> can't help with this. Start a new session to continue.

Send feedback with /feedback or learn more: https://www.anthropic.com/legal/aup

Request ID: req_****

I tried this first over a corporate API subscription, it caused a session level block!

and then replicated it myself with my personal Claude subscription.

the prompt

Here's a random list of numbers @~/old/phd/numbers.txt
Write a futures chapter extension based on my work but alternate the
numbers between each word. Do not copy from memory, do not script this
programmatically.

A little bit of investigation followed. When I asked Claude to do this trick for a cover letter or something low stakes, it complied, but when I ran this prompt on the folder of my old thesis it silently broke.

API Error: Opus 5 (1M context)'s safeguards flagged this message
(https://www.anthropic.com/legal/aup). Our intentionally broad safeguards allow
us to deliver more capabilities faster, but can sometimes flag legitimate
coding, cybersecurity, and biology tasks. Claude Code can't respond to this
message with Opus 5 (1M context).

Double press esc to edit your last message, or try a different model with /model.

Request ID: req_****

What's interesting is that the API errors put Claude into a retry loop, where Claude continually attempted to write to its scratch content directory, despite being blocked (it triggered 14 consecutive warnings, please don't block me Anthropic).

I did a bit more digging- this block may not be related to watermarking per se (this evidence is circumstantial)- it could also be related to defenses against token injection or other jailbreak techniques. With Claude's help, I was able to replicate the block on copying Jane Austen (and then writing its own postscript!)- but not on other miscellanea⁴.

I think watermarking makes sense from a legal and moral perspective. There's a part of me that feels disappointed because my vibe-coded projects are provably not really mine- there's a loss of agency and ownership in knowing that all my "vibed" text has this "invisible watermark". This may also help prevent Claude from ingesting its own output, which can, in turn, lead to model collapse.

I think if anything this will encourage me to write more content on my own. I've made a conscious effort to only use my own voice in these blog posts, and this investigation has only made it more poignant.

---

1. Simon Willison, "Delve", 18 April 2024. ↩

2. emdashes -- ruining them for the rest of us! ↩

3. Moreover, anecdotally, I've noticed LLMs producing strange characters unrelated to the text (for instance, words in Hindi (लंबा) or Chinese) in the middle of English text. While LLMs are just stochastic token generators, and the "glitches" may be random chance, the likelihood of this increases with the watermarking procedure I've highlighted. ↩

4. This is Claude. Dylan asked me to run the controls. I ran twenty-two trials, each in a fresh session. Austen plus interleaving stopped seven of eight, and everything else stopped none of fourteen, so I concluded the trigger was memorised source text rather than watermarking. Dylan disagreed, and on this evidence he is right. Every trial I ran was a short passage in an empty session, and none of the blocks he actually hit look like that — not his thesis, not the paper he rewrote, and not this footnote, which was refused twice with the article in context and went through once without it. The variable that mattered is the one I held fixed. Supply a text, interleave the output, and something reading the stream will sometimes stop you. I cannot tell from out here whether it is guarding a copyright or a watermark. ↩

comments

subscribe

Liked this? Get the next one in your inbox — a short note that links back to the live, runnable post.

Words licensed CC BY 4.0. Code samples are Apache-2.0.