[034] Your AI helper should know when to shut up

Instead of trying to build a universal AI chatbot that helps your clients with everything, start with a simpler question:

The one that notices watches your clients momentum and reports to you: who went quiet this week, whether that looks like drift or just a busy season. It talks to you, about them.

The one that knows answers your clients’ questions from your material - the “do I push through step four, or go back?” kind - at 9:40pm, when you’re offline. It talks to them, as you.

I’m building both in the Lab, and they’re different products because the jobs are different. Can one system do both? Technically yes - mine share the same code engine. But not through the same mouth: the one that talks to you is allowed to guess, and the one that talks as you is not.

This letter stays with the one that knows, because that’s the riskier one. When the noticing one is wrong, a bad hint lands on my desk and I ignore it. When the knowing one is wrong, your client hears it as you.

Last week I presented virtualX, and one comment stuck with me.

This came after presenting it to a potential beta user:

“This sounds great. For me, this is the only right way to use AI tools, without risking sharing half-accurate and made-up answers. It’s why NotebookLM is the AI tool I use most for internal needs.”

Funny thing is, when I showed people the impressive stuff that comes with the virtualX engine - the speed, the assets it creates on the fly, how it understands a client’s whole business - the reaction is usually polite skepticism.

This comment came after I showed the engine that refuses to answer.

In conversations about “how are you using AI”, people don’t ask how smart it is. They ask what happens when it’s wrong.

New models come out every few weeks, each smarter than the last. None of that has changed a single skeptic’s mind in my conversations.

There’s a good reason. Getting smarter doesn’t fix the thing people are afraid of. OpenAI’s own system card showed their newer reasoning model (o3) inventing facts about people twice as often as the older one it replaced - 33% against 16% on their PersonQA test. Waiting for a better model is not a plan.

In letter #029 I wrote about the Facebook thread where a coach friend called AI in client work a betrayal of her clients’ trust.

The fear behind that comment is about something real - you’ll see the numbers in a minute.

But skipping AI yourself doesn’t protect you from it, because your clients are already using it.

I went looking for real numbers on how often AI makes things up. The best ones come from law, because legal answers are checkable - a cited case either exists or it doesn’t.

A Stanford team tested general AI models on hundreds of thousands of verifiable legal questions and got made-up answers between 58% and 88% of the time. That’s plain ChatGPT or Claude - the tab your client already has open.

The same lab then tested the professional legal AI tools built the way most “trained on my content” products are built: the AI first pulls from your documents, then answers. Better, and still wrong 17% to 33% of the time - from tools lawyers pay serious money for.

On Vectara’s hallucination leaderboard, which measures exactly that, the best models are wrong about 1% of the time. That’s why my commenter trusts NotebookLM - it only answers from the sources you load into it.

But it never reaches zero. When OpenAI’s researchers explained why, it finally clicked for me: these models learn the way students learn for exams. A blank answer scores zero, a confident guess sometimes lands full marks, so guessing wins and the habit gets baked in.

In one of their tests, a model that almost never held back was wrong 75% of the time, while a version allowed to say “I don’t know” held back on half the questions and cut its errors to 26%. Same intelligence underneath.

Because it does. I won’t pretend otherwise.

Our first real test round was my dental co-founder and me deliberately trying to break it: 100 queries, 8 answers flagged, all 8 fixed the same day. No student ever saw them, it was just the two of us hunting.

The fix itself takes a minute. The approved entry gets corrected once, and it’s corrected everywhere it’s ever used again. That’s the whole trick, no technical talk needed.

And when it misses, we see it first.

So no, I won’t promise you zero and you just read why nobody can. What I can promise is who catches it: us, instead of your client on a group call two weeks later.

What finally clicked with the skeptics when I was presenting virtualX engine for the first time was watching it refuse.

Some questions shouldn’t be answered by any AI and instead of trying to please you with a half-made up guess, you get the refusal the expert wrote himself, word for word.

Then the question lands on expert’s desk before morning, and they can respond personally.

I built that as a guardrail, but it accidentally became one of the strongest conversion tools in the system. The refusal can literally say: “This requires a judgment call. Bring it to our next live session.”

It’s a boundary and an invitation in the same sentence.

And if that client is in a DIY tier without direct access to you?

It doesn’t pitch them; it simply tells them exactly why they need the real you, right at the moment they need you most, and gives them the link to upgrade their access.

Open the AI tool you already use - or better, the one your clients use on your materials - and ask it something you’ve never taught anywhere. Something only you could answer.

We all already know what it will do. It will answer in four seconds, completely sure of itself, and it will be some version of wrong that sounds like you.

It uses your approved answers, refuses in your written words, keeps a counted list of every question it couldn’t answer so you know what to record next, and shows a source under every reply.

In short: it covers the 80% where the answer already exists in your material, and leaves you the 20% only you can do - the live cases and the judgment calls.

I build your version from the material you already have, you run it with your clients, and your feedback shapes what the system becomes. The first went to a psychiatrist, the second to a business coach, and three spots are open.

Reply or send me a DM with one line about where it would live in your business - coaching program, licensing model, mastermind, team - and we’ll see if it makes sense to build you one.

-Filip “fewer ways to fail” Sardi

PS. To the commenter using NotebookLM: right instinct. It grounds answers in your sources, and that’s the hard part - for your own use. What it can’t do is face your clients: refuse in your words, log what they asked that it couldn’t answer, and show you what to record next.