Can an LLM have taste? Inkhaven Week 1, ranked by Claude

The obvious answer to the headline question is “yes” and then the followup question of whether that taste is good, and most people would immediately say “no”. Like many things LLM-related, I find myself thinking “wouldn’t it be cool if it could do this thing though?”

Inkhaven is going on right now, with 55 people in the cohort posting one blog post per day. Assuming all of them make it to the end, that’s 1650 blog posts! This is clearly an unreasonable number of posts for anyone to read, and definitely a case where some kind of algorithm, tagging system, or sorting is necessary. The Inkhaven blogroll allows residents and advisors to read and upvote posts, but this is subject to many of the same unpleasant pressures that e.g. reddit face, where anything upvoted gets more eyeballs and, in theory, more upvotes. Normally you would deal with this issue using confidence intervals or something, judging “read” against “upvoted” but there’s a limit on how much work I’m willing to ask Ops for.

So what if we got Claude to tell us what the best posts are?

In this particular case, the upvotes are hidden from Claude, so we do have some check of whether or not Claude’s taste aligns with the taste of Inkhaven generally, and perhaps this will surface great posts that, for one reason or another, no one read. I’m much more interested in the general problem of giving an LLM some notion of taste, which seems like it would be enormously useful, conditional on that taste being functionally useful to end users like myself, but this is a toy example. Thankfully, blog posts are mostly text, and we’re not going to worry about pictures. For simplicity’s sake, I’m limiting it to the first week of posts, all 445 of them.1

One of the benefits of this dataset is that all these posts were authored after the cutoff date, so none of them will have appeared in the training data; we’re getting Claude’s honest opinion, not the accumulated public opinion that we normally get. They’re also all written and posted across a single week, not that this should affect much. And I’ll be feeding these into Claude one at a time, without any hints as to metrics.

Of course, most of what we get out of Claude is going to depend on what our prompt looks like. One of the immediate pitfalls we want to avoid is a tendency toward the mean, as well as sycophancy; the LLM should use the full scale of 1-10 scoring. But one of the other things you can do is to have a rubric for rating where you split out things that might be important; this will help the LLM be more swingy in its scores, since it won’t want to bunch up scores around the 6-7 range for every one of these.

I settled on Interesting, Original, Well-Written, Informative, Funny, Insightful, and Overall (with specific instructions that it doesn’t need to be all those things in order to score a 10 Overall).

Here are some charts!

You can see that these scores are not well-distributed. There were, across all these metrics, 3185 chances for Claude to give a 10, and it gave 6 of them. I gave it instructions that roughly 5% of the posts should get a 1 and 5% should get a 10, and it didn’t follow that at all … but of course, I was having it read in each of these without the others in context, so on reading any particular post, it wouldn’t be able to know how that post stacked up against its peers.

We can also see that not all of these metrics are judged equally. Many more of these posts were judged as “not funny” than were judged “not interesting”. But I consider this to be evidence that there is real taste at play. The Inkhaven cohort is extremely varied, with fiction alongside non-fiction, serious investigative science posts and personal diary entries. Most people aren’t trying to be funny! So for Claude to say “actually, most of these aren’t funny” is a sign that these ratings are reflecting something I think is true, rather than standard bias.

By this naive method, these are the top 10 blogposts from Inkhaven Week 1:2

1. The Chimera, Thessaly Blue

2. An Open Letter to 21 people I know who died, Layla Hughes

3. Greatness, Soter

4. I’m Not Here, smorgasb.org

5. Every Lighthaven Writing Residency, Layla Hughes

6. Inkhaven Day 7 - Alone in an Airport, Anna Mattinger

7. A 500-word practical guide to getting enlightened through getting your legs waxed, specifically, Natalie Cargill

8. The Super Mario Galaxy Movie doesn’t care about rehabilitative justice, Kuiper (Alumni)

9. “Love, Mum” — What AIs can’t see about abuse, Natalie Cargill

10. Shrine of Sloth, conq

These are, perhaps surprisingly, not too different from the posts that were most highly upvoted by Inkhaven residents and advisors! These aren’t the same as the list of the most upvoted posts, but there’s pretty significant overlap (also, we’ll know more once Inkhaven is over, since most of the residents are focused on writing, not reading). There are posts I think should be there, ones I read that I liked, but I’m not offended by Claude’s taste.

But this is just having Claude naively rank posts, with no other context on what peers are doing. There’s got to be a better way!

A better way of doing things that would elicit more “taste” from Claude would be to rank two posts against each other. We could take each post, do a pairwise comparison against every other post, and then at the end, have a list of victors or other ranking system based on these pairwise matchups. This is classic voting system technology, well-known, and where there are discrepancies (A beats B and B beats C but C beats A) there are ways of hammering it out.

Unfortunately, doing pairwise comparisons on this dataset would mean 103,285 API calls. There are ways to reduce that, because not every pairwise comparison gives you an equal amount of information, and you can do caching stuff to reduce costs further … but it would probably still be too expensive for the purpose of this blog post.

So instead, I put the posts into a tournament, five to a batch, and had Claude rank which of these was best. This meant that each post was put up against a random subsample of its cohort, and the scores were necessarily distributed; Claude had to take a stand. To mitigate the issue where a great post might get matched up against an even greater post, this tournament was randomized and run three times.

Here are the top 10 pieces by this method:

1. Three Stones are Enough: The Case Against Leaves, in Particular,3 Anna Mattinger

2. An open letter to 21 people I know who died, Layla Hughes

3. endometrial biopsy, kaylee

4. Softhead, macroraptor

5. Every Lighthaven Writing Residency, Layla Hughes

6. The largest manufacturer of feelings in human history, Natalie Cargill

7. The one that loved me most, MLL

8. I did it. I found the worst poem in the world., Natalie Cargill

9. “Love, Mum” - What AIs can’t see about abuse, Natalie Cargill

10. Lost Mesoamerican Technologies, Lost Futures

We see lots of recurring names here, both from the previous list, and within this list. Claude really seems to favor Natalie Cargill and Layla Hughes, even more so than when giving direct ratings. Why? Difficult to say, but this might be something embedded in the model, a way that both of them are signaling quality through their particular voice. Maybe they’re two of the best writers! They’re certainly activating Claude’s neurons in some way that other writers aren’t as much.

I think, as a method of surfacing the best posts, this … might actually have been okay? That is, I’m not about to start trusting Claude as a recommendation engine, but I do think that a list like this is better than scooping up a random handful of blog posts. Maybe, as a thing that sits alongside the upvote system, it actually might have some value?

Claude clearly has some biases, and some of these are due to how it was prompted. It would generate a different list of posts if I told it to behave in a different way, and here, seems to rate literary/personal works higher than, perhaps, I would.

Could it be possible that LLMs are at the point where specifying your tastes can surface things you’ve never heard of that you’d be more likely than average to enjoy? From this, I conclude … kind of? The total cost for this blog post was $6.70 in tokens, which I was more than happy to pay in the name of science, but is perhaps not what I would want to pay in most other circumstances.

So far as I know, LLM taste-making is a long, long way away from being useful for people, but some of this is just because of integration and cost. As a webfic author, I do wonder whether there might be enough demand to set up a scheme to separate the wheat from the chaff, but I definitely don’t want to live in a future where I miss out on any delicious wheat because it was too outside the range of Claude’s tastes.

There is, every day, a torrent of content being produced, and current methods of filtering that content kind of suck. This isn’t the silver bullet, but maybe it’s a bullet I want in the chamber nonetheless.

Claude rates this post:

• Interesting: 7

• Original: 6

• Well-written: 6

• Informative: 5

• Funny: 3

• Insightful: 6

• Overall: 6