When you add AI to your product, you might notice customers struggle to trust and use it. The problem is often the product decisions leading to the AI addition.
This week, we have a guest post from Patrick Neeman. He has been designing AI experiences as a senior Principal UX designer and in various leadership roles for over 10 years. Patrick has been incorporating gen AI into enterprise applications at Workday for the past 3 years.
Patrick will share eight decisions a product manager must make from discovery through launch and operation before adding AI to a product.
Most AI failures are organizational, not technical. Ironically, it’s usually a failure of the human in the loop making unrealistic decisions. I’ve spent 11 or so years around AI, the last three and a half in generative AI. The bottleneck sits upstream every time: in the human-created scaffolding around the model, not the model itself.
Most of it is just solid product and software practice, but now more important than ever.
Eight problems, each wearing a technical hat. Here’s each one with the human decision it’s really asking you to make.
Most AI efforts start from the technology solution and work backward to a use case. The model can summarize, so we add a summarize button. It can chat, so we bolt on a chatbot. Nobody asked what the person was trying to accomplish; the technology proposed itself, and the team said yes.
This is a failure of the most basic product principle: what are you solving for?
We should always start from intent: what problem the person is trying to solve before any mention of AI. I’ve written about what Will Manidis calls a tool-shaped object. That’s something that produces the feeling of work without the work. Capability-first AI is a factory for them: it feels productive, looked great in the demo, and ships nothing anyone needed.
If you can’t state the job and the outcome plainly, you need to do more work. Otherwise, the most capable model in the world will execute the wrong task perfectly.
What product managers should do
Write the intent as a user goal. Use their words, with no mention of AI. Bring UX and research in before the solution is chosen.
Kill capability-first ideas. If the pitch leads with what the model can do rather than what someone needs, send it back until it names the job.
Walk into most AI initiatives and ask what success looks like. You’ll hear improve productivity, or reduce costs, or, more candidly, leadership wants AI. None of those is an outcome. They’re wishes with a budget attached and they can’t be measured, which means they can’t be defended when the spending review arrives.
Marty Cagan has argued for years that teams should commit to business outcomes, not system output. AI makes this even more urgent: decide what changes in the business if this works, name the number, and you’ll know whether to scale it or kill it. Product teams decide which problem is worth solving, and the model won’t do that for you.
What product managers should do
Write the outcome as a number. If you can’t state the metric that will move, you’re not ready to build.
Pick a job, not a category. “Cut first-response time in support” beats “add AI to support” every time.
The fastest way to fail with AI is to reach for the most powerful version of it without understanding the need.
Fit the tool to the job: an agent when a decision must be made, automation when the steps are fixed, an assistant when someone just needs an answer.
For example, chat fits exploration and turn-by-turn steering, but ambient agents can be a better fit for ongoing work. Ambient agents work well on background work, triggered by an event, surfacing only when it needs a decision or hits a limit you set.
List to decide whether the work is agentic.
A real decision under uncertainty. If you can draw the flowchart, it’s automation, not an agent.
Steps that depend on each other, so the plan adapts as it goes rather than following a script.
It has to act, not just answer — calling tools, not only returning text.
Inputs too varied to hardcode, where a fixed workflow would break or need constant patching.
What product managers should do
Match the tool to the task. Decision needed → agent; fixed steps → automation; retrieval → assistant. Work with engineering to challenge if an agent is really needed.
Match the mode to the moment. Chat for exploration; run ongoing work ambient, on triggers, not behind a chat box someone has to remember to open.
Demand proof of autonomy. If a vendor sells an agent, ask what it decides without a human. If the answer is nothing, it isn’t one.
A model knows what’s in the prompt and what’s in its training. It does not know your pricing exceptions, your renewal calendar, the customer who threatened to leave last week, or the unwritten rule that finance signs off on anything over ten thousand dollars.
The MIT researchers found that pilots stall on brittle workflows and a lack of contextual learning, and the tools never absorb how the work is actually done. It’s a human context-supply problem, and the supply chain runs through your documentation, your data, and your willingness to expose both.
A mediocre model with rich, current context will outperform a frontier model working blind.
What product managers should do
Inventory what the model can see. Write down the context it has and the context it needs; the gap is your roadmap.
Surface the context it can’t reach. A document no system can retrieve is context the model doesn’t have. Naming those is how you know where the risk is.
Ask three people to summarize the customer feedback and you’ll get three different documents; ask a model and you get a fourth. Since the instruction never specified length, audience, format, what to include or leave out, you get different answers.
Ambiguous in, ambiguous out.
The model fills any gap you leave with a plausible prediction. It predicts likely next words rather than running your instructions literally, so controlled, unambiguous language is a core skill now, not a nicety. A request with clearly defined fields, explicit constraints, and worked examples beats a paragraph of hopeful prose every time.
Good product teams have spent years turning fuzzy intent into precise decisions, requirements, and constraints. AI just made that discipline foundational for everyone who ships.
A controlled prompt is built, not typed. I use a format I’ve written about — CARE: Context, Ask, Rules, Examples. The background the model needs, the request in one clear sentence, the constraints it must respect (format and tone included), and worked examples of what good looks like. Leave any of the four blank and the model fills it with a guess; name all four and the output stops drifting, because there’s nothing left to infer.
What product managers should do
Write the prompt in CARE format. A blank in any one of the four is where the output starts to drift.
Specify before you prompt. State audience, length, and constraints up front; don’t make the model infer them.
Most AI systems rarely tell users what to give them for a good result. A user types a vague line, gets a vague answer, and concludes the model is weak when the real gap was the context.
Telling humans what to give a system is the cheapest quality lever there is.
Likewise, a good system makes it clear what it can do and how well it does it. Most products hide limits, then act surprised when users get a wrong answer.
Accuracy itself has to be an expectation we set and check as humans in the loop by design. None of this is glamorous, which is why it gets cut, but it’s the difference between a tool people trust and one they quietly stop using because they never understood the constraints.
What product managers should do
Tell users what to give it. Show a good request, the context to include, and what the system ignores, so a vague input doesn’t become a vague output you both blame on the model.
Tell users what to trust. State what the system does well, does poorly, and how often it’s wrong. This makes an uncertain answer look different from a sure one.
Most AI deployments have no owner, and that’s a problem.
There’s always a sponsor who approved the budget and a vendor who sold the tool, but no one is accountable for what happens when it decides wrong. That vacuum is fine in a demo; it’s not fine when the agent has write access to customer records.
One of the most reliable ways to stop an agent from taking a harmful action? Assign an owner to it, and they set the rules to make sure it never has permission from the beginning.
You give AI the same guardrails you already give people. That’s true human in the loop.
Security teams have known the fix for decades under a plain name: least privilege. Give the system the narrowest access that still lets it do its job, and the exposure of any mistake shrinks to something you can live with.
What product managers should do
Scope permissions to least privilege. Everything the agent cannot reach is a harm it cannot cause.
Name an accountable owner and a stop condition. One person is answerable in production, and a documented way to halt it, settled before the first deployment, not after the first incident.
Most teams can’t answer a simple question about their AI feature: is it better or worse than last week? When they change the model, tweak the prompt, or swap a vendor, they’re guessing; the system might have improved, might have quietly broken, and nobody can say.
Evals are the fix: a fixed set of real inputs with known-good outputs, scored the same way every time, so a change produces a number instead of a feeling.
AI needs more test cases, not less, because the output is probabilistic and a change that helps one case can break another you never thought to check. Most AI teams have none.
The eval set is built from the messy real cases your users send, the edge cases that embarrassed you, and the categories that matter to the business. It’s the only thing that turns “the new version feels better” into something you can defend.
Without evaluations, every release is a vibe and every regression a surprise.
What product managers should do
Build an eval set from real cases — actual inputs, especially failures and edge cases, with the output you’d consider correct, and score every change against them.
Gate releases on the score. Don’t ship a model, prompt, or vendor change that drops the number, however good the demo looked.
The AI models are ready; the organization around them usually isn’t.
Human problems are the ones you can fix without waiting for a vendor. Start from the user’s intent. Name the person accountable. Give the model the context it can’t infer. Specify the work in language tight enough to execute. Tell the user what to expect. Measure whether it works. Define the outcome as a number you can defend. Match the tool to the job instead of buying the most powerful one in the room.
Do all of this and the model you have performs like the one you were waiting for. It was our expectations of technology that were still early in its maturity. The hard part of any technology was always going to be us and that we can control.
Many thanks to Patrick for his best practices from working with agentic AI. If you enjoyed this article, I encourage you to check out Patrick’s newsletter AI for UX. Patrick’s insights will help you navigate the complexities of user experience with confidence and creativity.
Patrick turned these eight decisions into a practical AI Product Readiness Checklist you can use with your team before an AI feature ships.
Download the AI Product Readiness Checklist
It’s designed to be something you can bring into a product review—not another AI framework to read and forget.
Product Management IRL free resources
The GenAI Divide: State of AI in Business 2025 (MIT NANDA) — ~95% of enterprise GenAI pilots deliver no measurable P&L impact; the barrier is a learning gap, not model quality.
Claude Code turned every engineer into three (VentureBeat) — the bottleneck shifting from writing code to deciding what to build.
Gartner: over 40% of agentic AI projects canceled by 2027 — escalating costs, unclear value, inadequate risk controls, and “agent washing.”
An AI coding tool wiped a company’s production database — the 2025 Replit incident; the fix was enforced dev/prod separation.
Guidelines for Human-AI Interaction — Amershi et al., Microsoft Research; the first two guidelines are about setting expectations.
Automation Bias in AI-Assisted Medical Decision-Making — experts overturned their own correct calls to follow erroneous AI in ~7% of cases.
The State of AI in 2025 (McKinsey) — a defined process for validating AI output distinguishes high performers.
Outcomes Are Hard — Marty Cagan (SVPG), on moving from output to outcomes.
From Model-centric to Data-centric AI — Andrew Ng, on why the fastest gains come from improving the data.
The key to great AI-generated content — the CARE prompt framework.
Agentic Content Design — Yuval Keshtcher on content design as rule-governed language.
Tool-Shaped Objects in UX, Part 1 & Part 2 — building from intent rather than capability, and what survives the shift to AI.
Connect with Amy on LinkedIn, Threads, Instagram, and Bluesky for daily product management insights.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.