We are in a strange moment with AI-assisted development. The projections keep going up. Anthropic’s Fable 5 system card suggests AI can make development 4X faster. Opus 4.1 claims 2X. Earlier papers put it at 1.5X. The direction of every headline is the same: AI tools make developers dramatically more productive.
And yet, a real study - 16 developers, 246 engineering tasks, a proper A/B setup - found the opposite. The teams using AI tools were 20% slower than the teams that didn’t.
I want to explain why that is, because I have been watching this play out firsthand with my team and with our customers. And more importantly, I want to share the approach that is actually working.
Let me make sure the result lands clearly. This was not a benchmark. It was a controlled experiment - half the engineers got AI tools, half did not. Entry-level and mid-level engineers both. And when they measured actual task completion time across 246 engineering tasks, the AI-assisted group spent 20% more time.
A team of five engineers, given AI tools under this dynamic, effectively needs a sixth person to maintain the same output.
When I brought this to our community session, I asked the room: why is this happening? The answers were thoughtful. Old agile ceremonies slowing down AI-native workflows. Engineers spending time reviewing AI-generated code instead of just writing it. Security and non-functional requirements emerging late, forcing rewrites. Each of these is real. But they are all symptoms of the same underlying cause.
The root problem is that the way most teams use AI to build - give it a PRD, get an app back - is not a production workflow. It is a prototyping workflow. And prototyping workflows have different failure modes than production workflows.
Vibe coding - give a prompt, get an app - is genuinely useful. It is the fastest path to a mock-up, to early customer feedback, to seeing whether an idea is worth pursuing at all. I taught it. I still recommend it for those purposes.
The problem is not what it produces. The problem is what it leaves out.
When you give a two-page PRD to Claude Code, V0, or Lovable and let it generate an app, you get something that looks real and often behaves incorrectly. In the demo I walked through during the session, the vibe-coded contract review app had authentication that accepted any string - type “ABC” and you were in. When asked to find a clause on page three, it hallucinated a page reference that did not exist. The chat interface responded to “hi” with a fabricated legal finding.
This is not the model failing. This is an architectural gap. Four things are missing every time:
Production coding standards. If you do not specify your front-end framework, your API patterns, your folder structure, the AI chooses. It may choose Next.js when your codebase is React. Or it invents a structure that cannot be integrated with what you already have. Without these constraints, the generated code cannot enter a real codebase without significant rework.
Engineering team context. The AI does not know your existing databases, your existing schemas, your existing authentication system. So it creates new ones. Now you have three databases storing overlapping information, and your product analytics are split across tables that were never designed to talk to each other. The data you need to understand your product does not exist in one place anymore.
Telemetry and observability. Vibe-coded apps almost never include hooks for tracking what users do, what the AI costs per session, where errors occur. These are invisible at demo time and essential at production time. If you cannot measure it, you cannot improve it.
Design system coherence. The app your AI builds has buttons, colors, fonts, and interaction patterns chosen by the model. If you already have a product, the new page looks like it arrived from a different film - an action movie that suddenly includes a love song.
When I see teams struggling with this gap, they usually try one of two things before finding what actually works.
The first instinct is to add more prompting - iterate with the AI to fix the bugs it introduced. The problem: every iteration is an open loop. The AI rewrites large portions of code to fix a single issue, and the new code introduces new issues. There is no learning, no accumulation. You are spending tokens in a cycle, and the cycle has no exit condition.
The second approach - which has real merit in theory - is test-driven development. Write all your tests first, then give the tests and the PRD to the AI and let it build in a closed loop until the tests pass. This is better. But in practice, it has two hard problems. First, it is cognitively difficult for humans to write comprehensive tests before they have seen any working code. We are not trained to do it, and the tests we write tend to be incomplete or misaligned with the actual behavior we want. Second, running this loop is expensive - customers who have implemented it are spending 10 to 20 times more tokens than before.
The approach that is actually working is newer and takes a different inspiration: it mirrors how good engineering teams have always worked, with the addition of AI at each stage.
The core insight of spec-based development is simple: the reason vibe coding produces unreliable output is that the AI is being asked to make too many decisions at once. It decides the architecture, the module boundaries, the data model, the authentication approach, the UI patterns, the business logic - all in a single pass, from a two-page document.
The solution is to separate those decisions, make them explicitly, and review them before the AI writes a single line of production code.
Here is how the workflow runs.
Step 1: Write a production-ready PRD. The traditional PRD - problem, solution, user, metrics, roadmap - is still necessary but no longer sufficient. The new PRD adds four sections specifically for AI agents. First, your grounding strategy: which knowledge sources the AI should draw from, and how it should anchor its answers to that source (so “page three” means actual page three). Second, your hallucination constraints: what the agent should refuse to do, what it should deflect to a human, where its boundaries are. Third, your cost budget: how many tokens or dollars you are willing to spend per user interaction, per session, per task. Fourth, your evaluation strategy: what good output looks like, how you will measure accuracy, what the ground truth is.
These are not extras. They are the product decisions that make an AI-powered application function correctly.
Step 2: Generate a high-level design.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.