RSS Amplifier

Artificial Impact · Aug 18, 2026

Why your AI POC looked great (and why it failed in production)

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Alexandre du Sordet · Artificial Impact

AI generated

Hello everyone,

Picture a steering committee, nine months ago. A document-processing POC hits 94% accuracy on the demo set. The vendor’s slides are immaculate, the live demo lands every example, and someone actually applauds. Green light, budget unlocked. Last month, the project was quietly shut down. In this article we are going to deep dive into why AI POCs that demo brilliantly still die, and here is the twist we will unpack together: most of them did not break in production. They were never telling the truth in the first place. By the end, you will have a catalog of the five ways a POC lies to you, and a gate checklist to catch each lie while it is still cheap. Let’s go.

If you have ever visited a developer’s show apartment, you know the trick. Perfect light, rented furniture, no neighbors, and the plumbing is not connected. You fall in love with the model unit; you get the unit on the third floor facing the parking lot.

A POC is a show apartment. In our composite case (details blended from real projects, as always), the 94% was measured on PDFs the vendor’s own data scientist had selected: clean scans, one template, no handwriting. Production meant twelve templates, phone photos of crumpled invoices, and a label that quietly encoded the answer. The demo was not an early version of the product. It was a different product that happened to look identical.

A few weeks ago, we covered the 5 things that break when your AI leaves the lab: drift, cost, monitoring, ownership. That article was about what happens after launch. This one is the prequel: the failures that were already baked in before launch, visible during the POC itself, if you knew where to look!

ℹ️ Small glossary

  • Leakage = information from the answer sneaking into the training or evaluation data, inflating scores. The model “cheats” without anyone intending it.

  • External validation = testing a model on data from a different site, time period, or population than it was built on.

  • Wizard-of-Oz demo = a demo where humans quietly do part of the work the system claims to do.

  • Shadow deployment = running the model silently on live traffic, comparing its outputs to reality before letting it act.

  • Class imbalance = when the interesting cases are rare, making “accuracy” a misleading score.

So... how does a demo lie while everyone watches? In five well-documented patterns.

A student who aces a home exam may simply have had the answer key on the desk. Leakage is exactly that: the model sat the exam with the answers in the room, and everyone applauded the grade. It comes in flavors: features that are proxies for the label, preprocessing fit on the full dataset, duplicate rows across train and test, and the deadliest for business cases, temporal leakage, where a model is validated with knowledge of a future it would not have at decision time.

This is not a fringe problem. Princeton’s Sayash Kapoor and Arvind Narayanan documented leakage-driven overoptimism in 294 published papers across 17 scientific fields. Those papers were peer-reviewed; your POC review meeting is gentler. Zillow is the industrial cautionary tale: its pricing models looked accurate on historical data, but buying houses forward is a forecasting problem, not a curve-fitting problem. When the market turned, Zillow Offers shut down with a $304M inventory write-down and a quarter of the workforce cut.

Epic, the electronic-health-record giant used by many of the largest hospital systems, shipped a sepsis-prediction model. Models like this are graded with a score called AUC, and the idea behind it is refreshingly simple. Pick, at random, one patient who truly had sepsis and one who did not; the AUC is the probability that the model gives the genuinely sick one the higher risk score. An AUC of 1.0 means it never gets that pairing wrong; 0.5 means it is just flipping a coin. Epic advertised 0.76 to 0.83, which sounds solid. Then Michigan Medicine checked it on 27,697 of their own patients and measured 0.63: in real use it missed two-thirds of actual sepsis cases while crying wolf on 18% of everyone admitted. Same model, different data, completely different truth.

IBM’s Watson for Oncology, the system that was supposed to bring AI to cancer treatment, went further: it was trained largely on a small set of synthetic cases from one hospital. MD Anderson, the Texas cancer center that piloted it, spent $62M before shelving it, and internal IBM documents described “unsafe and incorrect” treatment recommendations. If your POC’s eval data was curated, single-source, or synthetic, you have not evaluated your system. You have evaluated your curation.

A chef can plate ten perfect covers for a restaurant critic. Friday at 8pm, with 200 covers, two no-show line cooks and a broken fryer, is a different restaurant. Demos are tasting menus: ten hand-picked examples that work. Production is Friday service, where the tail of the distribution walks in. McDonald’s learned this when its AI drive-thru pilot ended in 2024 after viral order fails; Taco Bell scaled back its own in 2025 after a prankster ordered 18,000 waters. For LLM POCs there is a subtle variant: benchmark contamination, where the model has literally seen the test during pretraining, or the team unconsciously tuned the prompt against the eval set. The open-book exam again, in modern clothing.

Some pilots work because humans quietly do the hard part. Amazon’s Just Walk Out stores famously relied on human reviewers behind the scenes. Plenty of agency demos have the same safety net; it just is not on the slide. A human-scaffolded pilot proves you can deliver a service. It does not prove you have a system. If the scaffolding cannot be priced into the production plan, the POC proved the wrong thing.

“94% accurate” sounds decisive. Here is the tiny worked example that should hang in every steering-committee room. Suppose 6% of your 1,000 monthly invoices are the fraudulent ones you care about:

Same accuracy, possibly zero value. The metric has to be the one the business bleeds by. Amazon’s recruiting model faithfully optimized “resembles our past hires” and learned to penalize the word “women’s”. Klarna, the Swedish buy-now-pay-later giant that went AI-first on customer service, celebrated an assistant “handling two-thirds of chats”, until the metric that actually mattered (customer experience) forced a public reversal and re-hiring of humans in 2025.

Europe has its own monument to the wrong metric. In 2020, with exams cancelled, England’s regulator Ofqual used an algorithm to hand out A-level grades. It was tuned to keep each school’s results in line with its own history, which it did beautifully, while downgrading nearly 40% of teacher-predicted grades and hitting talented students from poorer schools hardest. It triggered street protests and a government U-turn within days. The model optimised the wrong thing, matching the past, instead of the thing that actually mattered: judging each student fairly.

You have seen the numbers: S&P Global found 42% of companies abandoned most of their AI initiatives in 2025 (up from 17%), with about 46% of POCs scrapped before production. RAND puts AI project failure at up to 80%, twice the rate of ordinary IT projects, and its #1 root cause is misunderstanding the problem the AI should solve. That is not a production failure. That is a POC-design failure, discovered late.

And the famous “95% of GenAI pilots show no ROI“ from MIT? Handle with care. The same report found roughly 90% of employees using personal AI tools daily; the 95% measures formal pilots with measurable P&L attribution, in a small, non-random sample, and critics pushed back hard. Failure to measure value is not the same as failure to create it.

Now the counter-argument, because it is half right. A POC is supposed to cut corners; its job is to kill bad ideas cheaply. Scrapping 46% of POCs might be the system working. A 100% survival rate would be the real red flag. The sin is not that the POC used clean data. The sin is letting “the demo worked” stand as evidence for a claim the POC never tested, and then skipping the gate between demo and production. Two honest caveats before we build the checklist. First, the famous disasters usually had more than one cause: Zillow also made a bad business bet (it kept winning the bidding precisely because it was overpaying), and Watson was sold hard long before it was proven. A deceptive demo was necessary for these failures, but rarely the whole story. Second, be a little skeptical of the failure statistics themselves, because the loudest “everyone is failing” narratives are often told by the consultancies and vendors selling the cure. Trust the direction these numbers point in; verify the exact figure before you quote it.

Here is the checklist we use, sized for a steering committee rather than a research lab. We did not invent it; it distils two well-known references. Google’s ML Test Score is a 2017 rubric engineers use to check whether a machine-learning system is genuinely ready for production. REFORMS is a 2024 checklist agreed by 19 researchers for running machine-learning studies without fooling yourself. We have simply shrunk them to boardroom size; items 1 to 5 cost meeting time, not headcount.

  1. Provenance ⇒ who selected the eval data and demo examples, and what was excluded? No answer means tasting menu.

  2. Leakage audit (did the model secretly cheat?) ⇒ Did any input quietly carry a hint of the answer? Did the team prepare the data (cleaning, scaling) using the test set as well as the training set, which is a classic accidental leak? For anything time-based, was the model only ever tested on dates that come after the ones it learned from? And for an LLM, could the test questions already be sitting inside its training data?

  3. Ugly-data eval ⇒ run the POC once on unfiltered, recent, worst-quality inputs. Decide in advance what score drop kills the project.

  4. Business metric before demo ⇒ define the KPI the CFO cares about and the conversion path from model metric to KPI, before anyone claps.

  5. Human-scaffolding census ⇒ list every human intervention during the pilot, with its production price tag.

  6. Shadow phase before scale ⇒ silent deployment on live traffic with pre-registered success criteria. This is the bridge to the previous article’s territory: shadow mode is the last pre-production gate; monitoring is the first post-production one.

  7. Time-travel backtest ⇒ for anything forecasting-shaped, hold out a later time period including a regime the model never saw.

For enterprise teams, one addition: make the gate a formal stage with a named owner who did not build the POC. External validation needs external validators.

  • Most “production failures” are evaluation failures discovered late ⇒ the cheapest fix in AI delivery is asking harder questions at the demo, not buying more MLOps after it.

  • Leakage is common enough to distort science (294 papers, 17 fields) ⇒ assume your POC has it until an audit says otherwise.

  • External validation routinely halves vendor claims (Epic: 0.83 claimed, 0.63 measured) ⇒ never green-light on someone else’s eval data.

  • Accuracy is not a business case ⇒ a 94% coin can catch zero frauds; define the money metric first.

  • POCs should die often ⇒ the goal is not more surviving POCs, it is POCs that tell the truth about what they proved.

The one-line takeaway: a POC is not a small version of the product; it is an argument, and your job at the gate is to check whether the argument survives contact with reality. You are allowed to love the show apartment. Just do not sign the mortgage until you have seen the real floor, on a rainy day, with the neighbors home.

Thanks for reading Artificial Impact! This post is public so feel free to share it.

Share

What was the moment you realized a demo had lied to you?

Research

https://reforms.cs.princeton.edu/

Industry / news

Read the original on artificialimpact.substack.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.