Your Product Is Never Finished

If your product has an agent or a chat interface at its core, it behaves differently every time a user touches it. Same question, different user, different result. The input is unpredictable, and so is the output.

You can’t fully test this before you ship. You can write all the unit tests you want, but they’ll cover the parts of your system that behave predictably - which is no longer the part that matters.

The product is built in production now.

Not because you’re being reckless - that’s just where reality is. Your users’ actual inputs, their actual workflows, their actual edge cases - these only exist in production. You can simulate them, but simulation and reality diverge fast when a language model is making decisions.

This means your development process isn’t “plan, build, test, ship” anymore. It’s SOAR: Ship, Observe, Adjust, Repeat. Not monthly or bi-weekly at the end of some Scrum sprint - continuously.

When I built my last agent, the prompts I wrote on day one were reasonable guesses. After two weeks of real user data, they were actually effective. The difference wasn’t cleverness - it was information that only exists when real people use the real thing

This changes three things most companies haven’t caught up with:

How fast you go live. If your first release takes more than a few weeks, you’re already behind - not because of competition, but because every week in development is a week without data. Your assumptions aren’t getting better in staging. They’re getting stale. The first version doesn’t need to be complete, it needs to be real.

What “done” means. Nothing is done. Your agent’s performance is a curve you’re constantly shaping, not a box you check. A feature isn’t finished when it passes code review, it’s finished when it reliably handles what your users actually throw at it. And since that changes, “finished” is a moving target.

How you think about quality. With traditional software, you could theoretically cover every input and output before shipping. With AI products, you can’t - the space of possible interactions is too large and too unpredictable. The only way to know if it works is to ship it and find out. Quality isn’t tested in. It’s iterated into - live.

Why most companies resist this

Because every instinct in an established software company says: control the release. Test thoroughly. Don’t ship until it’s ready.

For AI products, “don’t ship until it’s ready” becomes “never ship” - because it’s never ready.
And for many companies, it becomes “never start” - because the bar feels impossibly high.

The risk isn’t shipping too early. It’s optimizing in isolation for so long that your assumptions go stale, and your competitors learn things about the market that you can’t, because they were in production while you weren’t.

Ship. Observe. Adjust. Repeat. That’s how products are built now.