AI Feature Development & Evaluation Services for Growth-Stage SaaS Companies Consulting services offered as low-risk, fixed-price packages. If you don’t f...
Here is an LLM request written with the Anthropic SDK the way the rest of the TypeScript world would. If this feels alien, then I suggest starting with this tutorial , where I break the call down in plain TypeScript first. const client = new Anthropic ({ apiKey : process.env.ANTHROPIC_API_KEY ! }); const message = await client . messages . create ({ ... }); Observe thy mark of exclamation. That is…
The problem with using LLM Judges is that it's hard to trust them. If an LLM judge rates your output as "clear", how do you know what it means by clear? How clear is clear for an LLM? What kinds of things does it let slide? or how reliable is it over time? In this post, I'm going to show you how to align your LLM Judges so that you trust them to some measurable degree of confidence. I'm going to…
One big barrier to testing prompts systematically is that writing evaluations usually requires a ton of setup and maintenance. Also, as a TypeScript engineer, there aren't that many practical guides on the topic, as most of the literature out there is for Python developers. I want to show you how write your first AI evaluation framework with as little setup as possible. What you will need LLM API…
In early 2024, I ran an AI-enablement workshop with iFeel , a company that provides online mental health and wellbeing support for employees, to help them understand how tools like ChatGPT can be utilised at work. My approach was to review all the official sources of information on prompt engineering at the time ( OpenAI's guidelines , Anthropics' , free courses , some paid ones ). I then…
The whole point of building an evaluation suite is so that you can systematically improve your AI product. Without evaluations, your only option is to respond to problems as they arise. As users complain about your app doing something unexpected, you rush to fix it. When you try to fix things, your only option is to test the improvement out on a handful of examples. You never really know if it's…
In a recent presentation at AIE, Rafal Wolinski and Vitor Balocco shared how they set up evaluations for Zapier. Given how new evaluations are for everyone, I thought it would be interesting to examine the approach Zapier’s took and see how it compares to how I run Evaluation, based on what I learned from with Shreya and Husain. I'm interested to see what the differences are, where they present…
To visualize this, we have a basketball court. Blue represents shots made, and red represents shots missed. The first property to consider is that the farther away your shot is from the basket, the harder it is to make. Another property is that the court has boundaries. So this blue dot—although the shot goes in—is out of the court. So it doesn’t really count in the game. Let's say you built an…
I’ve been reflecting on what it means to become a better prompt engineer. The field is expansive—prompt engineering for RAG applications? Better conversational interfaces? Building reliable agents? What does becoming a better prompt engineer even mean? People talk about prompt engineering like a bag of tricks—the emphasis is on the special phrasing, templates, and intricate formatting. Staying…
I just watched a presentation by Ankur Goyal at the AI Engineer's World's Fair, where he flips the idea of Evals-are-unit-tests-for-ai on its head. Typically, we build a feature, then write some tests to protect it. When your tests fail in the future, you know you've broken something and need to fix it before pushing any changes. This is true of conventional code as much as it is for evals. Goyal…
I'm working through Hamel Husain and Shreya Shankar's, AI Evals For Engineers & PMs course at the moment. I wanted to process and share my current understanding of how to evaluate the AI bit when you're building an AI feature. First instrument your application Instead of manually checking every AI response, you need a way to review thousands of responses quickly. There are tools for this (I've…