Build, evaluate, and improve AI agents that work in production.
Save 25% ($1,050) on September's cohort with this link.
---
🏆 The most comprehensive AI evals course available. We've refined this material with over 4,500 engineers and PMs from teams like OpenAI, Google, Meta, Amazon, and Microsoft. Read what students say →
📚 View the syllabus here
More than a course: an ongoing system, tools, and community for shipping better AI.
🎮 A private Discord community for ongoing support, even after the course.
🤖 6 months of access to our AI Evals assistant.
♾️ Lifetime access to all recordings, materials, and future cohorts.
💬 10+ hours of live office hours to get your questions answered.
Do you catch yourself asking any of these while building AI applications?
How do I test outputs that need subjective judgment?
If I change the prompt, how do I know I am not breaking something else?
Where should I focus my efforts? Do I need to test everything?
What if I have no data or customers? Where do I start?
What should I measure, and what tools should I use?
Can I automate evals, and how do I trust it?
If so, this is for you. All sessions are live and recorded.
📝 We've completely refreshed the material for our Sep 2026 cohort. Scroll down to see the syllabus.
What you’ll learn
Build a real AI agent, find where it breaks, and improve it with evals you can trust, working the full loop hands-on.
Learn directly from Hamel & Shreya
Hamel Husain
ML Engineer with 20+ years of experience.
Previously At
Shreya Shankar
ML Systems Researcher Making AI Evaluation Work in Practice
Shreya builds systems for AI-powered data processing. She holds a PhD in Computer Science from UC Berkeley and, in 2027, will join Carnegie Mellon University as an Assistant Professor of Computer Science. Shreya created DocETL, an open-source system for analyzing unstructured text at scale, deployed across journalism, law, medicine, policy, finance, and urban planning. Her research has been published at top computer science venues including VLDB, SIGMOD, and UIST (including a Best Paper award), and has influenced systems at Snowflake, BigQuery, LangChain, and OpenAI. Before her PhD, Shreya worked as a ML and data engineer at startups. She holds a BS in Computer Science from Stanford University.
Shreya & Hamel are co-authors of the forthcoming O'Reilly book Evals for AI Engineers
What's included
Live sessions
Learn directly from Hamel Husain & Shreya Shankar in a real-time, interactive format.
Your 24/7 Evals Assistant (6 Months)
Stuck at 11pm on a judge prompt? Ask the AI we built from everything we teach. Students only.
Lifetime Access: Recordings, Materials & Every Future Cohort
Rewatch anything anytime, and rejoin any future cohort live at no extra cost
150+ Page Course Reader
We provide a course reader with detailed notes to supplement your learning and act as a future reference as you work on evals.
Lifetime Access To Discord Community
Private discord for questions, job leads, and ongoing support from the community (over 1000+ students and growing).
10+ Office Hour Q&As
Open office hours for questions and personalized feedback.
4 Homework Assignments With Solutions & Walkthroughs
Optional coding assignments & walkthrough videos so you can practice every concept.
Certificate of Completion
Share your new skills with your employer or on LinkedIn.
Course syllabus
17 live sessions • 11 lessons
Schedule
Live sessions
3-5 hrs / week
Lectures are delivered live but also recorded so you can watch the materials at your own pace. We also provide over 10 hours of office hours and a community where you can ask questions (even after the course ends!).
Sat, Sep 5
6:00 PM—7:00 PM (UTC)
Wed, Sep 9
3:00 PM—4:00 PM (UTC)
Wed, Sep 9
4:30 PM—5:30 PM (UTC)
Optional Homework Assignments
1-2 hrs / week
Optional coding homework assignments where you implement evals from scratch. We provide all students with solutions and associated walk-throughs.
A good place to start on error analysis with Lenny Rachitsky
A one-hour conversation on error analysis.
See what our students have to say
See more testimonials at https://bit.ly/eval-reviews
More Testimonials
See more reviews at bit.ly/eval-reviews
Can PMs also get value from this course? Yes! This is what PMs are saying:
.png&w=1536&q=75)
https://x.com/ttorres/status/1933296711658815722
6 Months Access To An AI Evals Assistant w/Everything We've Said re: Evals
This is a special tool for students and not available for sale. Experimental and for learning only.