Evals in Himalayas : Inside AI's Global 'Evaluation Gap

I am writing this from a wooden table in Himachal, at an altitude where the wifi comes and goes like it has somewhere better to be. There is a dog outside who has adopted me for the week, a cup of chai going cold next to my laptop, and somewhere below the fog line, an entire mountain I still haven’t figured out how to describe without sounding like a postcard.

I am also, if you can believe it, reading benchmark papers up here. Not because anyone asked me to. Because I am the kind of person who cannot just use a model, I have to know why it does what it does, and lately that rabbit hole has led me somewhere unexpectedly funny: it turns out almost nobody agrees on what a benchmark score even means.

Let me back up. I am not a developer. I have never shipped code that mattered to anyone but me. What I am is a person who loves sitting in front of a big, complicated model and poking it from different angles, permutation after permutation, mostly just to see what falls out. So when I say I’ve been thinking about evals, I mean it the way a curious amateur means it, not the way an ML engineer means it. Which, weirdly, might be exactly why this topic is fun to write about. I have no ego riding on the answer.

Here’s what sent me down the hole. I kept seeing builders online say some version of “we switched models because ours dropped on the leaderboard,” as if the leaderboard was a mountain elevation chart you could just read off and trust. And then I found this piece from Epoch AI that quietly wrecked that assumption for me. They took the same benchmark, GPQA-Diamond, ran it across different providers serving the same open model, and the scores moved. Not by a rounding error. On agentic coding benchmarks like SWE-bench Verified, just switching the scaffold, the software that actually operates the model, caused an 11 percent swing for GPT-5 and a 15 percent swing for Kimi K2 Thinking. Some providers were quietly returning empty or truncated responses, some hit rate limits, one had a token cap so low it couldn’t even properly run the eval. None of this shows up in the tidy little leaderboard number everyone screenshots.

Share

Sitting up here, that landed differently than it might have on a Monday morning at my desk. Because a mountain doesn’t lie about its elevation. A benchmark, apparently, absolutely can.

So then I went looking for why this happens, and found a piece from O’Reilly by Aishwarya Naresh Reganti and Kiriti Badam that gave me the sentence I’d been missing. They draw this line between the model layer and the product layer. The model layer is what OpenAI or Anthropic tests before they ship, general capability stuff, coding, reasoning, trivia. The product layer is your actual thing, the one with your users and your weird edge cases and your specific definition of good. Their line, and I’m going to butcher it slightly, is basically this:

benchmark scores tell you what a model can do in general, they tell you nothing about whether it works for what you’re building. And apparently so many builders fall into what they call model selection hell, endlessly comparing leaderboards, that it’s basically a recognised syndrome now.

I felt this like a personal attack because I have most definetaly picked a model because of a chart.

And then, because the universe rewards a girl who reads too much, I found the story that made the whole thing click.

Taylor Hughes , who cofounded a video generation platform called Hypernatural, wrote about realising that none of the eval tools out there, the fancy hosted ones, the open source ones, could handle a pipeline that moves across text, images and video all at once. Every tool assumed a straight line from one input to one output. His system was closer to a tree, branching, chained, one bad prompt rippling into a dozen bad frames downstream. So he built his own eval tooling, inside his app’s Django admin of all places, unglamorous, boring, completely his own.

My favourite line from the whole post, and I promise I’m quoting the bare minimum here because he earned it: “turns out Greg Brockman was right (assuming you have the right tooling).” And this one, which I think about constantly now: the road to hell is paved with, in his words, “Sick Demos” that took hours of cherry picked regeneration to fake.

Share

That’s the whole thing, isn’t it. A demo is a mountain photo taken from exactly the right angle at exactly the right hour so the haze doesn’t show. An eval, a real one, the kind you build yourself because nothing off the shelf understands your specific weird pipeline, is what tells you what the mountain actually looks like on a random Tuesday with clouds in the way.

I don’t have a clean moral for you here, mostly because I’m still figuring this out in real time, from a table with patchy signal, like everyone else. But if there’s one thing worth carrying into your week, it’s this: the leaderboard number is not your product’s number. Nobody’s benchmark, however official looking, was built for your exact use case, your exact users, your exact way things go wrong at 2am while some developer is are “on call” anxiously waiting to go to sleep.

If you’ve never run even a tiny, ugly, ten-prompt eval on your own inputs, this is your nudge. You don’t need permission or fancy infrastructure. You need ten real examples from your actual product and the discipline to look at the outputs honestly, the way Taylor did in his Django admin, the way I’m trying to look carefully at the fly that fell into my now not so warm cup of tea.

More soon, hopefully with better wifi.

PS: If you are a builder, developer or writer in the ML space and want to share some of your own product insights , hurdles you have faced and things that you feel people aren’t covering enough. Write to me at shreya@aiplus.dev, I would love to know more.

Love,

Shreya