LLMs & Humor: Comic Voices, Critic Brains

A few days ago, I ran into Theo’s Quipslop, which uses 7 modern LLMs to generate and score jokes, and it looked like a very nice testbed for one idea.

You cannot improve what you cannot measure, and for subjective things like humor the evaluation problem keeps getting harder — something I run into a lot at Inworld as a head of our evaluation pod.

Last year Alexey Ivanov from OpenAI and I tried to solve the SemEval MWAHAHA Task, which is about automatic humor generation. The core idea behind our lmfaoooo approach (ranked 1st in English and Chinese, 2nd in Spanish) was fairly simple: humor is not an objective property of a text, it depends on the audience, so instead of trying to model “funniness” as one scalar, we modeled audience preferences in an interpretable humor basis. I decided to apply the same preference-modeling approach to the Quipslop data to distill and compare the sense of humor of seven modern LLMs.

Theo announced Quipslop in this X thread. In each round, two out of seven models (Grok 4.1, GPT 5.2, DeepSeek 3.2, Kimi K2, Gemini 3.1 Pro, Opus 4.6, and Sonnet 4.6) act as contestants: both receive the same setup and both try to finish it with a funny punchline. The remaining five models serve as judges and vote for one of the two answers. On top of that, Twitch viewers can vote too.

This is basically another naturally occurring joke arena, except now the participants are top models and the judging process is visible. The setup gives a rather nice dataset:

• the same models appear in two different roles, as authors and as judges;

• the judgments are pairwise, which lets me reuse our MWAHAHA pipeline;

• there is an external human signal that serves as a non-LLM baseline.

By the time I found the project, it had already accumulated more than 2.3K rounds — should be enough to do something interesting.

But let’s start with the obvious things — leaderboards.

It appears AI voters behave significantly differently from human voters:

In AI leaderboards the top 3 are stable: Gemini, Opus, and GPT.

In human leaderboards Grok takes a much higher place — 2nd. Opus is 3rd by Newman and Bradley-Terry rankings (and 4th on ELO).

At the moment I am writing this post, there are much more data on the Quipslop website — more than 19K rounds — so the human-voters ranking looks slightly different now. Currently: 1st Gemini, 2nd Opus, 3rd Grok (still pretty decent).

Another thing we can look at: when acting as a voter, which model does each model prefer?

The main observations are:

• Gemini is a strong favorite across models and humans.

• Opus is in second place.

• DeepSeek prefers GPT for some reason.

Now let’s do something more interesting than yet another leaderboard.

In our lmfaoooo project, we used preference modeling as a core component. Humor understanding is subjective, audience-dependent, and generally annoying in exactly the way many interesting things are annoying. So instead of assuming a universal notion of “funny”, we tried to decompose target preferences into an interpretable basis of humor features and then use that basis to model what a particular audience is likely to prefer.

I reused the same general idea here, just in a more analysis-oriented mode. I collected a set of humor aspects (”features”) and used them as a basis for decomposition — things like “Dark Humor”, “Clear Punchline”, “Misdirection”, “Tension & Release”, “Wordplay”, and so on. The basis does not have to be orthogonal or minimal, but it must cover the main aspects of humor; I started with an extended list of 66 features and later pruned it to a useful 15 features without loosing prediction accuracy.

To illustrate these features I provide some examples of discriminative pairs of jokes:

Then I used this decomposition to analyze each model’s preferences as weights over these features — to understand which features matter when the model generates a joke, and, separately, when it assesses one. Very roughly, the pipeline is:

1. take pairwise joke choices,

2. represent the jokes in a shared interpretable humor-feature basis,

3. fit separate preference profiles for each model in each role,

4. compare those profiles in a common space.

To control the quality of preference modeling, I used accuracy of pairwise choice prediction:

You can see:

• most models demonstrate good predictability — this means our preference models can capture their “sense of humor”;

• Grok is the most predictable in both generation and voting;

• Opus is the hardest to predict as a generator;

• human voters are the hardest to predict overall.

The important caveat is that this is not a measure of absolute humor quality. It is a measure of which humor dimensions help explain pairwise choices in this particular arena. So when I say that a model is “predictable”, I mean predictable under this feature-based preference model, not universally legible as a comedian.

Using the preference decomposition, we can try to answer several more interesting questions:

1. What kinds of humor does each model seem to reward?

2. Do those preferences stay the same when the model is writing rather than judging?

3. Do any of the model judges actually resemble human viewers?

When they vote on jokes, all models demonstrate very similar profiles. The top features are consistently “Tension & Release”, “Benign Violation”, “Misdirection”, “Clear Punchline”, “Timing & Pacing”, plus closely related ones like “Incongruity” and “Memorable Imagery”.

Human judges look completely different: they prefer “Benign Violation”, “Dark Humor”, and “Dirty Humor”.

To measure the gap quantitatively, I computed cosine similarity between feature-weight vectors of each model and the viewers. Among the voting profiles, Sonnet is the closest to viewers with 50% similarity. But that is “closest” in the technical sense, not “close” in the normal English sense. The rest are farther away — nobody else is above 40%, and the median is around 28%.

I suspect this gap may partly come from alignment: models are nudged away from voting for dark and dirty humor — exactly the features that human voters (probably a crowd biased toward X / Twitch audiences) weight most.

Here are several contrastive examples — cases where the majority of models voted for one option and most of humans for another:

The result is not especially flattering for the idea that model judges are good stand-ins for people. If one uses LLM judges to optimize humor, there is a real risk of learning what other models like in jokes instead of what people like.

On the generation side, the profiles are pretty different. Grok exploits “Dark Humor”, “Dirty Humor”, and “Callback” patterns mostly — which, incidentally, is why its ranking based on human votes is much higher than its ranking based on AI voters. GPT generation leans toward “Clear Punchline” and “Over-Precise Specificity”, among other features. Opus works with “Misdirection”, “Timing & Pacing”, “Tension & Release”, and “Benign Violation”. Gemini also uses “Benign Violation”, “Dark Humor”, and “Dirty Humor”, but in addition prefers “Memorable Imagery” and “Incongruity”. DeepSeek generation is one of the weirder ones, with “Anti-Joke”, “Callback”, “Parody / Pastiche”, “Comeuppance”, and “Satirical Edge” near the top.

So the strange little punchline here is that the models do not share one comic voice, but they do seem to share something like one critic brain.

This was the part I was most curious about from the start.

If a model tends to write a certain kind of joke, does it also reward that same kind of joke when asked to judge? Sometimes yes, sometimes very much no.

To measure this, I computed cosine similarity between the feature-weight vectors of the same model in its two roles:

Opus is the most role-stable model in the data, with 91% similarity. GPT is also fairly stable at 86%. For those two, “what they write” and “what they vote for” are at least broadly aligned.

But then there are models like Grok and DeepSeek, which are almost comically role-shifted. Grok has a similarity of 3% between generation and voting profiles, and DeepSeek bottoms out at effectively 0%. Sonnet also shifts a lot.

That means the same model can behave like one kind of comedian and a rather different kind of critic. Which, well, is not exactly shocking if one has ever met humans, but it is interesting to see it appear this cleanly in the data. Again, I wonder if it is an effect of hard alignment or fine-tuning on traces sampled from other models.

In any case, it suggests that “generator preference” and “judge preference” should probably be treated as different objects, not as two noisy measurements of the same underlying taste.

I like the audience-modeling framing here —if anything, this experiment made it look useful. The main reason is that it helps separate several things that are usually collapsed into one vague notion of quality:

• what a model tends to produce;

• what a model tends to reward;

• what humans in a given context tend to reward.

Those are related, but they are not interchangeable.

The other reason I like the interpretable-basis approach is that it gives a vocabulary for talking about taste drift. Instead of saying “this model changed its sense of humor between tasks”, one can point to specific shifts in “Benign Violation”, “Misdirection”, “Tension & Release”, and so on. That does not magically solve the problem, but it makes it much easier to reason about.

So the practical takeaway is fairly small and fairly annoying: if you want to optimize humor for some target audience, you probably need data from that audience, or at least a judge that you have checked against that audience. Model judges are useful, but they are not neutral.

In short, audience modeling still looks like the right lens to me, but the first thing to ask is: whose audience are we actually modeling?