What Your AI Cannot Prove

I wrote the foreword to Al Nugroho’s new book. Here is the argument I put in it.

Al Nugroho has written What Your AI Cannot Prove: Governing AI in Regulated Industries https://leanpub.com/WhatYourAICannotProve, and I was glad to write the foreword. It is the executive companion to the AI Engineering Series that Wing Yan Lau and I have been publishing, which means it does the one thing our volumes cannot. Ours are written for engineers. His is written for the people who decide whether any of it gets built, approved and defended, and who are never going to work through a chapter on retrieval geometry before Thursday’s approval meeting.

What follows is the substance of that foreword, in a little more room than a foreword allows.

I spent the first decade of my career designing engines at Ford Motor Company, and that work leaves marks. You don’t judge an engine on a test stand on a mild afternoon; you judge it at twenty below, on fuel that is out of spec, in the hands of an owner who never opens the manual. Name the failure modes before you build. Treat the average as the least interesting number in the report.

I then spent years on the other side of the table, running model risk management at a large bank, working out what the institution could defend later — to an auditor, an examiner, a court or a customer who had been wronged. I am building again now, at MoDeVa.ai (https://modeva.ai/)and through Knowlytix (https://knowlytix.ai/). Both seats teach the same habit. In the meeting where a system gets approved, what settles it is never what the thing can do. It’s what you can show me.

Fluent output talks serious people into bad decisions because a model’s answer wears the same costume as an answer somebody checked. Al puts this better in his opening pages than I have in years of memos: the demo and the production failure are one behavior observed on two inputs. The assistant that impressed the steering committee on a Thursday and the assistant that cited an irrelevant policy passage six weeks later were doing the same thing. What changed was which letter arrived.

Which is why the frameworks many of us spent years building still matter but no longer reach far enough. Model risk management was built for a model — a documented functional form, an output you can benchmark against a realized outcome. An agentic system isn’t that. It finds the facts it will rely on, picks which tool to call, acts and writes to a memory that shapes the next case. The model at the center may be the smallest of your problems. Validation that covers only the model covers the wrong object.

A prompt is not a control. An instruction in a model’s context is a request, honored only if the model reads those words as policy every time, including the time a complaint letter contains a sentence engineered to talk it out of the reading. A rule written in code, running on the proposed action before anything takes effect, is a control. Firms are going to lose real money in the gap between those two sentences.

A model grading a model is not evidence. Asking a second model whether the first one told the truth trades one stochastic opinion for another and calls the result verification. Splitting a draft into its individual claims is fair work for a language model. Deciding whether the fee was $35 is not, and belongs to a comparison against your own recorded value, which returns the same verdict on every run and can be re-performed by a successor team a year later.

A folder of markdown files is not a memory. A pile of documents with a similarity index over it is cheap, readable and unable to do the one job that matters. The March note and the July note disagree, both match the query and the higher score wins, because nothing in the store knows the two conflict. Figures held as prose come back paraphrased, and $35 becomes $45 on a bad day. A citation names a document rather than a span an auditor can open. That is why we put a knowledge graph and the Geometric Memory System underneath: exact values recalled exactly, contradictions detected by the store instead of argued about afterward. A memory that cannot return a verdict leaves you asking one model to vouch for another.

A human in the loop is rarely the control you think it is. Reviewing fluent output at volume is a demand no organization actually meets, and the failures aren’t random. Gell-Mann amnesia describes it exactly: the reviewer catches the error in the field they know cold, which is the error the system was least likely to make, and waves through the judgment they cannot assess. Oversight ends up weakest where it is needed the most. If human review is your control, the gate has to verify that the review happened and on what evidence.

A benchmark is not how you choose a model. A leaderboard says how often a model was right on somebody else’s questions. It tells you very little about behavior on your policy corpus, under your adversarial customers, inside your authority limits. We never picked a supplier’s component from a headline horsepower figure. Ask instead for training-data provenance, results disaggregated rather than averaged into one number, an honest account of failure modes and binding notice before the model changes underneath you — as contract terms, not as requests made after signature. Then run your own suite, on your own facts.

A decision made on an average is a dangerous decision. Ninety-two percent means the system passed on whatever mix of cases happened to arrive, and changing the mix changes the number. An average also treats a clumsy but correct reply as equivalent to an unfair-fee complaint closed automatically instead of escalated, and those two errors do not cost the institution the same money. The tail carries the exposure, and the tail is small enough for a mean to hide completely. A committee that would never accept a mean as a summary of an operational loss distribution will accept one as a summary of an AI system, because this number arrived with a technology attached. That is what makes an engine pass its test stand and fail on a freezing day in Minnesota.

One theme in the book deserves attention at leadership level. It anchors its controls to reasoning rather than to citations, because regulations move, as the recent withdrawal of a long-standing supervisory standard showed. A firm that keeps a control because a regulator demands it has just been handed permission to drop it. A firm that keeps it because the reasoning still holds hasn’t. You learn which kind of firm you are during the incident, not during the policy review.

None of the six points is a complaint about the technology. Each has an engineering answer, and most of those answers already exist. Ask what your AI can prove rather than what it can do. Most of the time the answer will be very little at first, and every gap it exposes is a specification for the thing you should have built.

The pilot went well. It was never the question.

What Your AI Cannot Prove is the executive companion, written for boards, executives and risk and compliance leaders. The engineering is in the technical volumes, each of which stands on its own.

Beyond “Prompt and Pray” — governed agentic systems: typed actions, controlled tools, gates that run in code, human escalation that can be verified and replayable audit trails. The first and fourth points above, worked through end to end.

Beyond “Chunk and Pray” — trustworthy retrieval and memory: why top-k similarity is not relevance, how facts are held with exact values and span-level provenance and how a system abstains when it cannot prove the answer. The second and third points, in code.

Beyond “Ship and Pray” — testing agentic systems: designed experiments in place of benchmark averages, factors that change the presentation without changing the correct answer and attribution that names the responsible component rather than reporting a scalar. The fifth and sixth points.

Knowledge Graph Embeddings as Geometric Operators — the mathematics underneath, for readers who want the foundation rather than the recipe. TransE, ComplEx, RotatE and the rest of the zoo as configurations of a single relation operator built from rotation, stretch and translation.