I Built a Benchmark to Measure AI Slop.

“You’re absolutely right”

“Great question!”

“Wait a second, that’s a game-changer!”

If you use AI tools, you know the feeling.

You ask a simple question and get back something that opens with “Great question!” and ends with a six-bullet summary of what it just told you, peppered with words like “delve” and “transformative” and “it’s important to note.”

It’s not wrong exactly. It’s just... sloppy, and it reeks of AI.

I got curious about how bad this actually was across different models, and whether anyone had tried to measure it systematically. Turns out, not really, so I built one.

Introducing SlopBench.

For the purposes of this benchmark, slop is any language pattern that has become a fingerprint of LLM output rather than natural human communication. Things like:

• Opener slop: “Absolutely!”, “Great question!”, “I’d be happy to help!”

• Word-level slop: “delve”, “synergy”, “transformative”, “nuanced”, “paradigm”

• Filler phrases: “it’s important to note”, “deep dive”, “thought leadership”

• Reframe constructions: “It’s not about X, it’s about Y”

• Bullet points and markdown formatting for simple answers

• Excessive em dashes

The benchmark runs 53 simple, chat-style prompts against each model, things like “What’s the best way to learn a new skill?”, and checks each response against a curated list of 150+ known slop patterns across seven categories. A response is marked as containing slop if it hits at least one pattern. Deterministic evaluations, no LLM-as-a-judge.

The system message is just “You are a helpful assistant.”, so this is pretty close to raw model behavior out of the box.

The full results are live at slop-bench.vercel.app. Here are the highlights.

OpenAI models kill it on SlopBench. GPT-5.3-Codex comes out on top with a 1.9% slop rate, barely any slop across 53 prompts. GPT-5.4 (4.3%) and GPT-5.3-Chat (5.7%) follow closely.

Claude Sonnet 4.6 sits at 9.8%, which is respectable. Grok 4.20 Beta lands at 13.2%.

Notably, no Chinese models in the top 10.

DeepSeek V3.2 comes in last at 44.2%. Nearly half of its responses contain at least one slop pattern. Qwen3-Max (32.7%), Gemini 3.1 Pro Preview (28.3%), and the Kimi models (both around 26%) are also toward the sloppy end.

Gemini 3.1 Pro is the only American model amongst a sea of Chinese open-source models in the bottom five.

Anthropic’s most premium model, Claude Opus 4.6, has a 24.5% slop rate which is significantly worse than Claude Sonnet 4.6 at 9.8%, and worse than several much cheaper models. More expensive != less sloppy.

I ran a separate smaller local test specifically looking at bullet point rate and em dash overuse. Initially, these were included in the main benchmark score, but I literally had to remove them because basically every model uses one or the other in every response.

The bullet point rate was especially egregious, but the em dash rate was still mind-blowing.

I also decided to track cost and latency, mostly because I thought it’d be interesting, but also to include a matrix of cost versus slop rate.

SlopBench is an early, opinionated benchmark. The slop list is curated by me, based on patterns I’ve seen when using AI. It’s not exhaustive and it’s not perfect.

The benchmark is open source and the slop pattern list is explicitly designed to grow. If you have phrases or patterns you think should be in there, send them to me, comment below, or feel free to open PRs/issues.