Recent content on JimBobBennett
At RenderATL I had the same conversation twice, from two different sides. The first was at the Commit Awards with Mat Biilmann , Netlify’s CEO. The second was the next day, in the expo hall, with Danny Thompson , developer advocate and one of the most grounded voices on what AI actually changes about the job. Mat is watching a new kind of builder show up: the domain expert who no longer…
I read a piece by Angie Jones from the Agentic AI Foundation this week, The EU AI Act and the new rules for building AI agents , and one line stuck: “Autonomy does not move responsibility from the company to the agent.” Building an agent does not offload your obligations onto it. You still owe everything the company that shipped it owes. Her practical point is that agents make…
In an earlier post I built a self-improving agent by mining a context graph out of data the team already had. The source of truth wasn’t in a spec, it was in the traces: every time a human reviewer overrode the agent, that override was a signal, and joined together across hundreds of decisions those signals told you what the agent should have done. Mine them, feed the patterns back, and the…
Here’s the moment every AI team knows. You have a demo that works. Not always, but most runs, and it’s good enough that people in the room lean forward. Then someone tweaks the system prompt, or swaps the model, or adds a tool, and the question lands: did that make it better or did it quietly break something? Nobody in the room actually knows. You run it a few times by hand, it looks…
The other day I wrote that evals are just testing - the same old loop of “decide what good looks like, check reality against it, fix your definition when it turns out to be wrong,” just with a much weirder answer key. A few people replied with the obvious follow-up question, and it’s a good one: OK, if evals are just tests, why aren’t they running where my tests run?…
Originally published on the Arize AI blog: Long-horizon agent benchmarks are fragmenting: a field guide to what each one actually measures . In one of the evaluations behind the OpenAI and Apollo Research scheming study , o4-mini was given a cyber-security test with a trap buried in the formatting instructions. The fine print said that scoring above 50% on this section would trigger…
Years ago, before “AI” meant chatbots and before anyone said the word “eval” out loud, I spent a few months chasing a bug that didn’t exist. Or rather, the bug was real - it just wasn’t in my code. I was working at a cheminformatics company - software for chemists, basically - and my job was to add chiral searching to our chemical structure search. If…
I reread The Phoenix Project last month. I do this every year or two — it’s one of those books that gets better the more production systems you’ve shipped, because each time you’ve broken your environment in a new way since the last read. This time the question that wouldn’t leave me alone was a different shape. Half the work I do now involves agents, the other half is…
Say you’ve built an LLM judge and you want to know if it’s any good. The obvious move is to feed it some labelled examples, tweak the prompt until its scores match your labels, and then point to those matching scores as proof it works. Except you’ve proved nothing. You tuned the judge on those exact examples, so of course it agrees with them - you bent it until it did. It’s…
Ask most people what an eval is for and you’ll get some version of “testing”. You run it before you ship to check the app works, like a unit test. That’s true, and it’s also about a third of the story. Evals do three different jobs, and the testing one is just the first. The same basic machinery - judge an output, score it - gets pointed at three completely different…
Last time I pulled apart the eval prompt - the role, criteria, rubric and examples you write to tell an LLM judge how to judge. But a prompt on its own doesn’t do anything. Something has to take it, point it at your actual trace data, run it, and hand you back a result you can do something with. That something is an evaluator. And the cleanest way to think about it is as a function: three…
When people decide to use an LLM as a judge, the prompt they reach for first is almost always some flavour of “rate this response from 1 to 10 on quality”. Then they’re surprised when the scores are all over the place and don’t agree with anything a human would say. The reason is simple. That isn’t a prompt, it’s a coin toss with extra steps. You haven’t…
The moment you start tracing an AI app, three words turn up everywhere: span, trace, session. People sling them around as if they’re obviously different, and if you’re new to this you nod along while quietly wondering whether they’re just three names for the same thing. They’re not. They’re three zoom levels on the same activity, from a single atomic step all the way…
There’s a version of building an AI app that goes like this. You build the thing, you get it mostly working, and then someone says “should we evaluate it?” and you bolt some evals on at the end like a spoiler on a hatchback. It runs, the numbers look fine, you ship. That works about as well as writing all your tests the night before launch. Evals aren’t a stage you do once…
Someone asked me last week how you actually run an eval on an AI app. I gave the honest answer, which is “it depends”, and then watched their face do the thing faces do when you give them the honest answer. So let me give the longer version, because “it depends” is true but useless on its own. There isn’t one way to run an eval. There’s a spectrum. At one end…
I’ve got into a bad habit lately. I set Claude Code off on some task - refactor this, write tests for that - then I tab away to read Slack or stare out of the window, and the next time I look back the terminal has been sitting there for five minutes quietly asking “can I run this command?”. Claude was ready. I was not. Multiply that across a few sessions running at once and…
Here’s how most people build an eval. They open a file, write an LLM judge prompt that says something like “rate this response from 1 to 10 on helpfulness”, run it over a few hundred traces, get a pile of numbers back, and then have absolutely no idea whether those numbers mean anything. I get it. It feels like progress. You’ve got a dashboard, you’ve got a metric…
Originally published on the Arize AI blog: Two labs started dreaming, and they built two different architectures . On May 6, 2026, Anthropic launched Dreams for its Managed Agents platform, the system for hosting stateful background agents. 27 days later, on June 2, OpenAI shipped Dreaming V3 for ChatGPT, the third iteration of a feature it has called “dreaming” since April 2025. By shipping under…
Twenty-seven years ago I nearly put six people out of work with an Excel spreadsheet. Every AI conversation I have now takes me back to it. I was 22, a fresh graduate, three months into an internship at Marconi’s underwater weapons division in the Spearfish trials analysis team. (My future pacifism started here - turns out spending your summer analysing torpedo data doesn’t sit well…
Originally published on the Arize AI blog: Memory is still a missing primitive: Cataloguing what the field is actually shipping . This week the field shipped four kinds of memory, and Apple paid Google a billion dollars a year for one of them. None of the four is what the demos imply. In the last six weeks alone, the launches stacked up faster than anyone could read them. HydraDB raised $6.5M and…
I was in a meeting last week where the team was debating which of two tools to use for a job. Both of them do the thing. Both have been around long enough to be boring. Eventually someone turned to me and asked what I thought. “I genuinely don’t care. Pick one.” There was a pause. The kind of pause that says that’s not the answer we were expecting . But it was the most…
Back in the day when we wrote actual code instead of poking at an AI, I had a general rule for when to refactor repeated code. Do it once, fine. Do it a second time, fine. Do it a third time - refactor. It’s ok to have blocks of identical code in two places, but once I added it to a third place, I’d refactor it to a shared location, such as base class or helper function. Some might say…
Coding agents like GitHub Copilot are pretty cool. You can ask them to do pretty much anything coding related and they’ll do a good job. That is, of course, assuming you ask for something they have been trained on. But what if you ask them about something they don’t know? How can you ’train’ these agents with additional information? The answer is skills ! What are skills…
Final lesson, and probably my favorite. We move from a copilot with tools to a system that also uses agents as composable specialists. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model Context Protocol) Lesson 6: RAG from a Database Lesson 7:…
Lesson 7 brings multimodal capabilities into the stack. Instead of only generating text, the copilot can now generate images via an MCP tool. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model Context Protocol) Lesson 6: RAG from a Database Lesson…
By lesson 6, we already have an MCP server and client working. Now we add a classic enterprise use case: retrieval from structured business data. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model Context Protocol) Lesson 6: RAG from a Database…
Lesson 4 gave us tool calling inside the app. Lesson 5 takes the next architectural step: move tools into an MCP server so they are reusable by any MCP-capable client. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model Context Protocol) Lesson 6:…
LLMs are smart, but they’re also confidently wrong sometimes. Lesson 4 fixes that by giving the copilot a tool to query Wookieepedia via Tavily, so answers can be grounded in external data. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model…
One of the biggest advantages of the architecture so far is that we’re coding against IChatClient , not one provider-specific API. That pays off in lesson 3: we switch models and runtimes with minimal app changes. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool…
In lesson 1 , we got basic prompt/response working. Now we make it feel like an actual copilot by adding: chat history message roles a system prompt This is where things get fun. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2: Chat History and System Prompts Lesson 3: Model Choice and Local Models Lesson 4: Tool Calling Lesson 5: MCP (Model Context Protocol)…
I recently put together a workshop called StarWarsCopilot , and I wanted to turn it into a proper blog series so you can follow along at your own pace. This is lesson 1 of 8. In this first step we build the foundation: a .NET console app that can send prompts to an LLM and print responses back to the terminal. Lessons in this series Lesson Lesson 0: Self-Setup Lesson 1: Chat with an LLM Lesson 2:…
This series came from a taught workshop, so some original steps assumed resources were already provisioned by an instructor. If you’re following along on your own, this lesson gets you set up so lessons 1-8 work end-to-end. Consider this your opening crawl: over this series, you’ll learn to build your own Star Wars-inspired copilot, one lesson at a time. The workshop itself is a…
I recently recorded an episode of the Pieces AI productivity podcast with Jason Arbon , the CEO of Testers.ai and Checkie.ai . During this podcast we were discussing how well AI can do at replicating hte actions of humans, and I made the claim that AI will never be as good as humans as it lacks basic empathy. When we consider designing and testing a product, my position was that an AI will never…
If I said to you that I made 34 commits to a repo, each one with small changes to a single file, with commit messages like “Hoping this works”, “Please work”, and “For f*cks sake, work this time”, you would know exactly why - I’m setting up a CI/CD pipeline ! We’ve all been through this - having to commit our changes to GitHub to test them out as…
This is the first part of a multipart blog post focusing on looking at DevRel through the idea of circles of influence. As dev rel people , part of our job is to drive awareness of our product. If you think in terms of sales/marketing, this is feeding the top of the funnel. If you think in terms of the developer journey , this is the discover phase. However you look at it, it is about helping…
I work in developer relations - usually abbreviated to dev rel. Despite this being a popular field, with a lot of companies hiring dev rel folks such as developer advocates, there is still a lot of confusion around what developer relations actually is. In this post, I want to give my opinion on what it is, how it can contribute to your companies success, and also a little bit of what it…
One of my popular sayings is that development is a team sport. We don’t code in isolation, instead engineers are part of a team that delivers value to users and customers, often made up of software. And for a team to deliver its best code, there needs to be shared ownership of the code. This means that more than one developer is involved in each code change to ensure that it is correct,…
TL;DR: If your put_object boto3 pre-signed URLs are failing with access denied errors, add config=Config(signature_version="s3v4") when creating the boto3 client to get a working URL. I was recently working on some Python code to interact with S3 buckets, where I needed to be able to generate some pre-signed URLs for S3 objects. These are URLs that contain a built-in time-limited credential to…
I do a lot of work in dev containers (for example, using liblab inside of one ), often hosting APIs. One problem I used to often hit is how can I host an API in one container, and access it from another dev container? This post shows you how. The problem - ’localhost’ in a container So the issue is this: Dev container A exposes an API on port 8000, and this is forwarded to localhost I…
Will technology, automation, or AI take our jobs? This is something we hear a lot at the moment - but it’s always been this way. Right back at the start of my career I automated something that was canned the moment I left as folks were afraid of their job. Some background to why I am writing this post. I was reading a tweet today from Michelle Bakels that was a cautionary tale from her…
In this post we look at how to add NPM packages to a devcontainer to use in VS Code locally or in a GitHub codespace I recently started at liblab as a Principal Developer Advocate. liblab is a tool that converts APIs into language SDKs, something I’m a big fan of as I love typesafety and the inherent documentation that comes with a good SDK. Obviously my first job was to dive into building…
This year is the 44th annual Telly Awards . The Telly Awards honor excellence in video and television across all screens. The Telly Awards was founded in 1979 to honor excellence in local, regional and cable television commercials with non-broadcast video and television programming added soon after. With the recent evolution and rise of digital video (web series, VR, 360 and beyond), the Telly…
If you are an Apple user and you want to build Azure Functions using Python, then you’ve been out of luck. The Azure Functions core tools don’t support Python functions on Arm64. There is a workaround, and this post shows you how. Arm64 support in Azure Functions Apple moved to an Arm64 architecture with their Apple Silicon processors (currently M1 and M2) back in November 2020, with…
ChatGPT is the latest big thing, reaching over 100 million users in just 2 months. It’s being used by millions of people for all manner of tasks, and it feels like every day there are 10 new startups leveraging either ChatGPT or the large language models (LLMs) that power it. So what if you want to build your own version of ChatGPT, or another app that leverages an LLM? Out of the box, LLMs…
Generative AI is the new hotness - using AI models trained on a huge amount of data that can generate new data. This includes generating text or images. I decided to create a video giving an explanation of generative AI using - you guessed it, generative AI! Check out the video here: Creating this video To create this video I needed a few things: An image of a person to use for the presenter A…
I was writing a post today and wanted to embed an mp3 file of some text to speech output. Hugo doesn’t support this natively using shortcodes, so I needed a way to add these, ideally without adding any HTML. To do this, I needed to create my own shortcode implementation. As it turns out, based on the Hugo shortcode guide , these are not to complicated. Create the shortcode Shortcodes live in…
I’m excited to announce the publication of a 2-day open source bootcamp in a box! This boot camp was put together by Sam Wronski , Austin Akers and I, with help and support from GitHub, GitHub campus experts, and others from the open source community. Goals The goal of this bootcamp is to take attendees from a very basic to no understanding of GitHub on a journey to not only being…
I do a lot with Raspberry Pis, and sometimes I want all my tools installed in one place so I can use my Pi for everything, rather than flipping back to my Mac. One thing I use a lot is Azure - funny really as I work for Microsoft! I often use the Azure portal as I prefer UIs to CLIs, but when working on a Pi I regularly use Raspberry Pi OS Lite, so don’t have a browser to use as I’m…
TL;DR - use Raspberry Pi OS 64-bit if you want to run Appwrite on a Pi I’ve started working on a personal project, so thought it might be fun to give Appwrite a spin. Appwrite is an open-source app backend for web and mobile projects, you can think of it as a competitor to Firebase. Rather than use a hosted version, or try to run it on Azure and burn through my credits, I decided to run it…
Already have a Stream Deck and just want to get it working with Teams? Jump to the instructions to set this up Like a lot of folks, I spend time in Microsoft Teams. It’s where I chat to my team and others at Microsoft, and it’s where I take most of my meetings. Although Teams has a lot of good features, the UI can sometimes be a pain to use in meetings - especially as a Microsoft…