How and why we added AI to our product

While many companies are rushing to bolt AI onto their products, AskCapy emerged from solving a genuine workflow problem we experienced daily. The idea naturally grew out of our work on the global search functionality in Atono.

We weren't trying to build a better keyword search. The goal was to answer the kinds of questions search is supposed to help with by synthesising real answers from your product backlog. Not just a list of results to sort through, but actual answers drawn from the unique data in your Atono workspace: stories, acceptance criteria, and comments. For everyone on the team—devs, PMs, QA, and support.

We’d already seen that keyword search had limits, so we started looking for better ways to make search genuinely useful. That’s when Lex Vorona, one of our senior developers, led the spike research into RAG (Retrieval-Augmented Generation)—a method that combines retrieval (searching for relevant documents) with generative AI to produce informed, context-aware responses.

The concept was straightforward: shortcut the research loop. Instead of searching for "email validation" and forcing the user to dig through dozens of stories and bugs, allow the user to ask "How do we validate email addresses?" and get an answer synthesized from all the relevant context.

This approach addresses something fundamental about how product teams work—the constant need to reconstruct context from scattered information across multiple stories and discussions.

AskCapy began as a focused spike—an initial experiment designed to determine whether AI could consistently provide accurate answers to specific product queries using only real backlog data.

We weren't building a full feature yet. This was about testing the basics with three key unknowns: viability, cost, and timeline.

• Can an LLM reliably answer real product questions?

• Is the accuracy good enough to use in a real workflow?

• What's the cost—both in time to build and runtime infrastructure?

• And how long would it take to get from concept to something we could ship?

Once we agreed on the contents of the spike, we used the time to build a small proof of concept—a chatbot powered by RAG.

That means instead of just using an LLM by itself, we paired it with a retrieval layer. We pulled real data from Atono—about a couple of hundred stories from production, including descriptions, acceptance criteria, and comments—and injected that into the prompts.

To test it, we used actual questions from our PM, like:

• "Can a story be in more than one timebox?"

• "How long can a bug stay in triage before it's stale?"

• "What happens when a user tries to add an attachment when they are out of storage?"

We scored the responses on three key factors: accuracy, specificity, and whether the answers referenced their source data. Accuracy averaged 2.9 out of 5, showing the model often provided reasonable but not perfect answers. Specificity scored higher at 4.7 out of 5, indicating that when the model answered, it was quite detailed and relevant.

On the cost and performance side, indexing the data and running around 100 question-answer cycles cost about $0.04 total—quite economical for early-stage experimentation.

AskCapy works differently from generic chatbots. When you ask a question, it searches through all your stories and bugs for relevant matches, then synthesizes an answer based on that specific context.

Unlike traditional search, that just returns a list of results, AskCapy provides both an AI-generated answer and links to the source stories for verification.

This dual approach—AI reasoning plus source citations—addresses the core weakness of AI systems: confident wrong answers. Every response includes links to the source stories, so you can verify the reasoning and dive deeper when needed.

Mark Chang, our senior dev who built the technical implementation, describes the two-part process: "We start by combining everything—item descriptions, acceptance criteria, extra content, and comments—into a single document and run it through an embedding model. Then we take the user's question, retrieve the most relevant backlog items, and pass both the question and those sources to the chat model."

The system also maintains conversation history for follow-up questions, but only within a single session. Start a new conversation, and Capy begins fresh.

One of the most compelling use cases emerged from our engineers: using AskCapy as a proxy for product managers when they're not available.

"If I'm stuck waiting on a PM to respond, I can use AskCapy, see what it suggests, and move forward based on that—then confirm with the PM later," explains Lex.

This eliminates a common development bottleneck. Instead of stopping work to wait for clarification, developers can get contextually-aware answers based on historical decisions, then proceed with confidence while keeping the PM in the loop.

For example, when working on a delete operation, a developer might ask: "Is this a hard delete or soft delete?" Capy can infer patterns from past stories and respond with something like: "In similar backlog items, entities were typically hard deleted—so it's likely safe to proceed with a hard delete here."

Ask Capy's applications extend beyond engineering teams to address common workflow bottlenecks across roles.

Product managers spend significant time fielding questions about specifications. "I get a lot of questions every day about what specific features are supposed to do," Tobias notes. AskCapy can field these routine specification questions, freeing PMs to focus on strategic work.

Support teams often need to determine whether reported behavior is expected or represents a bug. They have extensive product knowledge from customer interactions, but may lack access to original specifications that AskCapy can surface instantly.

New team members can get up to speed without monopolizing existing team members' time. Fresh hires can ask questions about domain-specific workflows and get answers grounded in actual implementation stories.

The feature even has potential for incident response. During high-stakes situations where teams need answers quickly about how the product is supposed to work, AskCapy can provide immediate context when the people who know the answers aren't available.

Once we closed the spike and moved to actually building AskCapy, we hit several challenges that taught us important lessons about implementing AI features.

First was the architecture complexity—figuring out how to access AWS services properly. There's significant complexity in setting up permissions so that our backend could securely call the models and services we needed without overexposing anything.

Second, we had to upgrade some of our libraries to keep pace with how fast the space is evolving. New models—like Claude Haiku, Meta Llama, and Amazon’s Nova—offer better performance, but staying compatible means updating tooling more often than most teams are used to.

Third, we learned that evaluating models isn't just about raw quality—it's about quality relative to cost per call. In some cases, less popular models outperformed big names when we looked at both answer quality and cost together.

We also discovered that prompt engineering is fundamentally different from traditional development. Our developers got to a point where improving prompts required different skills—more like directing an actor than writing code. The skills we needed were found in our documentation team. The analogy of a director working with an actor became particularly relevant: you need to provide clear context, motivation, and boundaries for what you want the AI to deliver.

We're continuously refining AskCapy to make it more helpful and reliable, focusing on two main areas:

Model selection and structure. We're exploring different models—currently using Claude Haiku, but testing others like Meta's LLaMA and Amazon's Nova to compare cost, performance, and context handling. We're also examining how data is structured, testing whether splitting information into smaller chunks might improve accuracy, and providing better metadata so models have clearer context about their sources.

Data Optimization. Turning a story or a bug into a search-friendly document requires some additional nuances based on how those items are written. As an example, if content uses a strikethrough font, is the intention of the author to effectively say the comment isn’t valid? If so, strikethrough content should be omitted from the retrieval process. And is the discussion on a story that happens in the comments area helpful in defining what the story does, or is it too meta, and the actual story’s intent is best gleaned from the other fields?

Prompt engineering. We're shaping how Capy responds so it feels like part of Atono—calm, thoughtful, and clear, rather than a generic chatbot. This involves setting clear intent, scoping answers, and establishing boundaries so the model stays focused and admits when it doesn't know something rather than guessing.

AskCapy represents one of our first steps into AI, not our last. On our roadmap are a number of features that help streamline common processes and reduce toil, e.g.: suggest the size of a story, suggest the risk rating of a bug, suggest acceptance criteria that might be missing, and detect duplicate bug reports.

But we're taking a pragmatic approach to adding AI to the product. We're using AI where it genuinely solves problems, integrating it naturally into existing workflows, and always providing transparency into how it reaches conclusions.

While our competitors have doubled down on AI agents for everything, we're more selective about where AI adds real value versus where it might complicate workflows unnecessarily.

The goal isn't to replace human judgment—it's to empower everyone on the product team to quickly find answers using the unique data in their workspace, reducing context switching and eliminating the bottlenecks that prevent teams from focusing on building great software.

Ready to see how context-aware AI changes the way teams access institutional knowledge? Try AskCapy in Atono.