cf #104: are we *really* saving that much time?

Hey there 👋

There’s this Italian proverb, fidarsi è bene, non fidarsi è meglio, that captures my current relationship with LLMs pretty well: trusting is good, not trusting is better.

We’ve all been sold the promise that a lot of work could be fully delegated to the Claudes and ChatGPTs of this world; but as we’ve been discovering, there’s a canyon between the promise and what we’re actually getting. And in that canyon is a vast amount of deliverables that look plausible on the surface but always, always need a few extra verification steps.

Here are three recent examples. You can look at them in isolation and declare them blips—though in my book, when the same blips keep happening, the proper name for them is patterns.

(if I sound a bit grumpy about this… I am)

I used Claude to analyse a .csv export from Search Console. Pretty standard stuff, except Claude fully ignored a bunch of pages without acknowledging it—and in one specific topic cluster, the absence of a single URL skewed the results entirely.

I caught it early because it’s my job to know Float’s commercial URLs and because I wrote and shipped the missing page myself. But someone less familiar with our website might not have noticed.

I was working on an outline for a different client and used Claude to find a few relevant stats and data points. I liked what it found, but mindful of the Italian proverb above, I clicked through to double-check.

The numbers on the website were completely different—and that’s where I discovered this particular model retrieves a cached version of a page, instead of the live one. This was brand new information to me, and something I only got to after several question rounds.

This may be a bizarre thing to be principled about in 2026, but I will always be the person who spends two hours reading & categorising every survey answer in a thousand-row spreadsheet manually instead of letting an LLM do it in 5 minutes.

One reason for this is I simply will not trust an LLM not to make sh*t up.1

I went through a 200-answer churn survey full of business-critical insight, broken down by the respondent’s company size. I knew that none of the companies in the sample had more than 100 employees; but when we ran the same sheet through Claude, it began attributing quotes and behaviours to a 100+ employee segment.

I don’t mind the fact-checking per se. Being rigorous at work is how we serve our audience and customers well. Accuracy is a point of pride, and making sure things are correct is a good use of our time.

What I do mind is that we keep being sold this story of a technology that makes our lives easier and saves us time so we can fully focus on better thinking and more strategic work… and yet here we are, with the time supposedly freed by AI tools now spent double-checking work done by the same AI tools, unsure if we can ever fully trust what we get out of them, and sometimes finding it so much easier to just do the thing ourselves.2 Like we used to.

It just gets tiring, you know?

1

The other more important reason is that there is a huge experiential gap between reading a report that states 50% of customers come to you because of X, and having to work through 400+ messages where customers mention variations on X and suddenly feeling their cumulative weight in your brain. “Saving time” should not come at the expense of “understanding nuance” → and yes, quite often the right way to understand nuance is to just sit in the data, row after slow and painful row.

2