What your users do when you're not watching

A successful mobility company I work with in Germany recently shipped their first AI agent - a chat interface that lets users query business data within their company.

They eagerly released the first version, let it run for a few days, and the bad news followed fast: All their users dropped out after trying it 2 or 3 times. This was a clear signal: Users were seeing absolutely no value.

The team had no idea why - they only got vague answers that the results were “somehow not satisfying”. And as it turned out, they had no way to even know what their users were doing, and how the agent reacted.

The first thing I did with them was set up tracing and evaluation using LangSmith. Now we could inspect the agent’s traces: every step it took, every output it produced, and - most importantly - every input its users typed.

And that - that was the real insight.

Sure, the team had tested their agent before shipping. But their developers were incredibly biased with their inputs.

Whereas the team assumed users would go in gently, asking simple questions such as “how high was our revenue last quarter?”, their users seemed to have other plans.

Among the first conversations I saw were gems such as “write a full business report for me as the CEO to steer my company” and “predict the number of orders over the next two quarters.”

Some of the messages were simply harmless banter (”Hi, how are you?”), But in most cases, it became evident that people simply had no idea how to formulate their questions in the first place or what the system was even capable of doing, usually setting their expectations way too high. And the agent on the other side diligently tried to help everyone and scramble together a plausible answer, which led to even more confusion. In some conversations, you could literally feel the desperation building on the human side.

In the old days of software, you could test every button before shipping. With AI at the core of the product, that’s impossible - the only way is to suck it up and iterate with real users.

We fixed the agent in under two weeks. Not by rewriting it - by reading what users were actually asking, and optimizing for them, mainly via prompting and context engineering. The solutions were right there in users’ inputs; they just hadn’t been looking.