Recent content on samek.fyi
Cattle not pets , for software developers: The shift from handcrafted servers to immutable infrastructure taught us that mutability is the sworn enemy of understanding. Any artifact that is edited in place creates drift. Drift is what makes systems impossible to maintain. Our ability to kill and regenerate infrastructure components is the reason we trust it. At Honeycomb, we kill the oldest Kafka…
I’m trying to get into a new habit. Previously, when I wrote some code, and it felt risky, I would reread the code more carefully, read the surrounding code, try it out locally or in staging, and monitor closely after deploying. When I write code with AI, I’m grateful to have the intuition that a particular code change is risky, since that represents years of experience with the types…
There’s an old business idea, usually attributed to Michael Gerber, about the difference between working in your business and working on your business. Working in the business is doing the thing. Working on the business is improving how the business works. I think this distinction works in an analogous way in software engineering. Writing code is working in the business. Everything around…
This is a list of AI benchmarks I’m watching. Last updated July 27, 2026. ALE-Bench ARC Prize Artificial Analysis Bullshit Benchmark CursorBench DeepSWE Design Arena Epoch AI Models EQ-Bench 3 EQ-Bench Creative Writing ExploitBench Frontier Code GameDevBench GBA Eval Gert Labs Rankings LiveBench LLM Debate Benchmark LMArena MathArena Mercor Apex METR Time Horizons RuneScape Bench SimpleBench…
I had an deck in Anki, a flashcards app, with an annoying problem that I had caused when I imported the cards. On some cards, the front of the card had text that linked to a URL. On others, the URL was only on the back, which meant I couldn’t click it when I actually wanted it. My first instinct was to export the deck to CSV, use AI to write a script to transform it, and then import it…
I’ve been doing a bunch of keeping up with and experimenting with AI. It’s still a surprise to me that I can create a Chrome extension in minutes, having never written one before, and having rarely programmed in JavaScript. I wrote a Chrome extension to copy page content, and another to send the same prompt to multiple LLMs, then a CLI tool to do the same with Simon Willison’s…
Whenever a new model is released, I read the benchmarks. I haven’t seen a good list of benchmarks. I tend to see them referenced and then bookmark them. Here’s my list. I keep a continually updated list of benchmarks at samek.fyi/benchmarks . I deleted the old list that used to live in this post so that I could link to the updated one instead.
I tried Dia , a new Chromium-based browser with built-in AI features from the creators of Arc . It has an AI chat sidebar that can reference any open tab, plus writing assistance that pulls context from multiple tabs. Their YouTube video shows someone writing an email in Gmail based on their Google Calendar tab. It’s a UX improvement over existing options. Brave’s Leo only works with…
Simon Willison released tool calling in his llm CLI. A tool is just a python function. Inspired by Coding agent in 94 lines of Ruby , I wrote a coding agent in Python. Here’s the code . aider helpfully explains the various editing formats people use for LLMs: whole returns an entirely new document. diff is a series of search and replace blocks. diff-fenced is like diffs but puts the filename…
Google released an impressive improvement to Gemini, which now tops the WebDev Arena Leaderboard . That position had been held by Claude for quite a while. Here’s a Pareto frontier for the Aider polyglot coding leaderboard . The analysis of March’s Gemini didn’t include reasoning tokens in the cost. They fixed the problem for May’s release. It’s too bad it…
Kagi benchmarks LLMs on a private benchmark. I asked Claude to build a Pareto frontier chart for Kagi’s April 17th, 2025, update ( source code ). Claude has the strange property of doing well on some benchmarks and not others. It creates half as much slop in longform writing as Gemini and OpenAI models, for example, and does well in a number of coding benchmarks, while doing much worse than…
I’m Brian Samek. I began my career as a sysadmin in public and university libraries. I’ve been at MongoDB for a decade, as Software Engineer, Lead, Director, and now Staff Engineer. I’ve helped scale our Continuous Integration system to handle hundreds of millions of tests a day, led a major software supply chain security initiative, and built teams that consistently deliver useful and…
I keep seeing descriptions of Model Context Protocol (MCP) that are either vague metaphors or get too specific too quickly. From the official docs : MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP…
Deep Research (three separate products! from Perplexity, OpenAI, and Gemini) is usually more than I want. Instead I want something that stops when it has the answer. Here’s a great example about predicting the next pope. It searched the web to find out what prediction markets said, aggregated and normalized the data, and gave me a chart of probabilities. Then it made a second chart…
A few days ago I wrote that Perplexity is better for quick searches. That may have changed. OpenAI announced some search improvements, though sources were not among them, and Sam Altman mentioned improvements to the model. ChatGPT now does a better job at my example query, explicitly noting that the two models have different behavior. I’m curious if this will be reflected on Search Arena .
o3 and gemini 2.5 pro being a single Elo point apart on LMArena with style control feels very steam-engine time . There’s an idea in the science-fiction community called steam-engine time, which is what people call it when suddenly twenty or thirty different writers produce stories about the same idea. It’s called steam-engine time because nobody knows why the steam engine happened when it did.…
Simon Willison writes that “AI assisted search-based research actually works now.” o3 and o4-mini agentic research is truly impressive, as are ChatGPT’s and Gemini’s Deep Research. But I consistently find queries where Perplexity gets it right, and ChatGPT and Gemini get it wrong. An example: “how is chatgpt-4o-latest different from gpt-4o.” Only Perplexity…