Artificial tongue twisters for TTS evaluation

We periodically release new TTS models at Inworld, and every new model needs testing. Tongue twisters make useful stress tests, but a model may already have encountered the classics during training. I used an agent to distill tongue-twister generation policies for five languages, then revised those policies through multiple rounds of generation and adversarial judging. In the final blind sample, the generated phrases nearly tied the classics on mean judge score, but it appeared to be another case of reward hacking. However, on its way, my system produced some pretty nice artificial tongue twisters. Check the collection at the end of this post.

I liked the idea of using tongue twisters for TTS evaluation since the moment it came to my mind, perhaps even a little more than the idea deserved. They are compact, easy to inspect, and deliberately awkward to pronounce.

How much wood would a woodchuck chuck if a woodchuck could chuck wood?

There was one problem. If a model heard a classic tongue twister during training, it may reproduce a familiar sequence instead of dealing with a genuinely new one.

So I hacked tongue twisters (no tongues were harmed.)

I first asked an agent to deconstruct how tongue twisters work (in English, German, Russian, Spanish, and Italian.) It turned those observations into the general generation policy (including language-specific rules). I call this approach policy distillation (I previously used a similar approach for humor generation.)

The first batch was funny but fairly easy:

Brave Bruno baked brown bread by bright brick bridges.

The tongue remained resolutely untwisted, so I started an iterative evolution loop.

My policy distillation here did not mean any fine-tuning. It was a loop around an explicit text policy:

1. inspect classic tongue twisters;

2. extract practical construction rules (the policy);

3. use them to generate new candidates;

4. compare a sample with classics and ask where the new ones still fall short;

5. patch the policy for the next iteration.

Later iterations treated every previous generation as a negative corpus. The comparison was done by a fresh-context judge agent based on randomly shuffled classic/generated A/B pairs. On the way, agents grew a small aux toolchain: tools to check novelty and build blind sets; to inspect phonetic structure, to model difficult combinations for each language, and look for recurrence and cadence.

And, as always, evolution eventually hacked the reward.

As I mentioned, first generations mostly produced too fluent strings. Classics more often alternate confusable sounds, move them between onsets and codas, collide them across word boundaries, or reuse a root in another grammatical role.

So, at some point, the policy stopped rewarding matching initials by themselves. It asked sounds to migrate through the phrase, change position, and return in forms that become unstable after fast repetitions.

Four loyal lawyers ruled, until four rural jurors overruled them.

The next useful change was an explicit automatic IPA pass between drafting and selection. The agent can do it manually, but it’s not always perfect, especially for hard non-English cases.

The IPA-based diagnostics looked for repeated or reversed phone transitions, near forms, consonant clusters, word-boundary collisions, and switches between related phones. Spanish b/v, English x, silent letters, dialect-dependent rhotics, Italian gemination, and Russian stress or palatalization can make anybody confused.

Wenn Riesen reisen, reißen ihre riesigen Reisetaschen.

Despite I specifically instructed the agent to avoid re-using known twisters, sometimes it cheated. It could replace every noun and still pour the new vocabulary into the same syntax shell.

Such “disguised descendant” items are not exact copies, but they reuse some of the elements: sound lattice, syntax, semantic scene, or punchline mechanism. For instance:

Generated: Which wrist strap slipped from the Swiss shop shelf?

Classic: Irish wristwatch, Swiss wristwatch.

It retains wrist, Swiss, and the /w–r–sw–ʃ/ interference. It may also borrow the question rhythm of “which witch would watch which watch?”

So, over the next few iterations, I had to expand the negative corpus beyond words. The agent began tracking lexical roots, semantic events, part-of-speech skeletons, phonetic mechanisms, rhyme patterns, and rhetorical devices. This caught many disguised rewrites.

Another generation: Which white witch switched the wide watch strap?

Another iteration separated three signals: structural IPA evidence, language-specific articulatory hardness, and oral quality. The judge chose the harder side as the stronger tongue twister in 26 of 30 pairs. Every generated winner was harder than its paired classic, and automatic hardness correlated positively with quality among the generated items.

Some harder generations still lost. Consider these two:

Thread thin silk through six thick sleeves.

Twelve texts vexed the sixth desk clerk.

They are physically awkward, but they are also short and not funny, just hard. Classics often add lexical recursion, meter, rhyme, or a comic second beat. Hardness alone is just not enough.

The final iteration added an oral-form score for changed-root recurrence, sound return, reversed bigrams, clause balance, cadence, and rhyme. Generation immediately learned the visible shape of the metric. In the last pools, the generator produced 90% of items in the same form, I called it “semicolon shell”.

The best-scoring synthesis was:

Strong lungs lengthen songs; long songs strengthen young lungs.

It looks good! It combines real /ŋz#l/, /ŋθ/, and /s#str/ collisions with a balanced return. The blind judge scored it 9/10. But…

Now, all the generations looked the same:

Boring. As it often happens with artificial evolution, the system had learned to reward the metric faster than the metric learned to mirror the natural diversity of classic tongue twisters.

Nevertheless, on its way to the hacked reward, my system was able to produce some pretty nice examples, so let me share some cherry-picked ones with you: