The Missing AI Evals Playbook

Beyond podcasting and bridging cultures, Stella’s day-to-day is deep in the trenches of AI product building — and for the past 2+ years, her focus has been on AI Evaluation.

We’re excited to share our latest work and invite you to our first Lightning Lesson: Build Your AI Evals & Analytics Playbook.

Reserve your free spot!

AI Evals has been the new buzzword in town, from OpenAI’s AgentKit release to AI startup CEOs sharing their takes on social media.

For me, this is more than just a buzzword.

Over the past two years, I’ve been deep in AI evaluation, designing frameworks, running real-world projects, and helping teams refine their AI products through data-driven iteration.

Here’s what I’ve learned: Most AI evaluations today are done wrong.

Much of what’s on the internet focuses on LLM-as-a-judge, i.e. using AI to evaluate AI. But the reality is there has been a growing divide between product builders and users:

• ⚙️ Engineers are content with automated evals

• 👥 Stakeholders and users don’t believe in them

Research has already shown that LLM-as-a-judge can be biased and unreliable. Yet for many teams, it still feels like the only option.

But is it really?

In our AI Evals & Analytics course (We offer 50% off discount code COCOONS for our Substack readers), we share a battle-tested playbook built from real-world projects. It combines human evals with LLM-as-a-judge to balance scalability and reliability, and brings clarity before release and confidence after launch.

And evaluation doesn’t end at launch. Analytics is just as critical. Once a system is live, you need to monitor its health, understand user behaviors, detect early warning signs, and turn data into insights that guide product iteration.

If you’ve been trying to design evals your team can actually trust, join us for a free Lightning Lesson on Maven. You’ll learn what truly matters in practice and how to connect evals and analytics into one continuous feedback loop that drives better AI products.

👉 Reserve your free spot on Maven: https://maven.com/p/8430db/build-your-ai-evals-analytics-playbook

Look forward to seeing you!