Recent content on Tom Hipwell
I’ve read a few of these summaries now and I think Simon’s (as always) is best so I’ll link it here. This is just insane. Wow. If you’d described this to someone in 2016 it would have been dismissed as sci-fi. I think it’s clear from this that uncontrolled, persistent and shared memory is a major risk - the message board pattern that repeats in this timeline and…
I’m a massive nerd but I thought this was a really fun post. These security tilted models are very aware and know how to interact with/exploit the evaluation harness, the juice is about halfway through the post: For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies…
I thought there was a bunch of interesting ideas in here as Peter walks through the L1/L2/L3 context design he’s used. In particular the L1 design is very neat. Formula aliasing (e.g. clever ways of reducing context size). Only using a single tool (matches my own experience, keep it simple). Style compression (formatting is meaningful, but group formatting and describe). The hierarchical…
Mario on session durability in Pi, a rough design pattern for resumable agents durability is multi-layered. for pi’s agent abstraction, durability is really just about: store completed LLM responses store completed tool execution results (only the parts that get send to the LLM, not the effects, e.g. file changes, separate concern) store phase we are in (idle, llm streaming, tools executing,…
The writing on the Claude blog is consistently great, and often becomes the defacto standard for the subject at hand. This post is nominally a marketing piece for dynamic workflows but the article also serves to collate a bunch of multi-agent interaction patterns in a single place. Each pattern is well named and simply described. There’s a note right at the bottom which gives the right…
Steering is the process of boosting model weights during a session - if you can work out the relationship between concepts and model weight activations (easier said than done) by examining how activations change with slightly varying prompts within the same session, then you can reinforce the concepts that matter most to you (“write clean code”). The article dives into why this may or…
I’ve been looking at how different agents are handling creds and while looking at the NanoClaw code I learnt that they refactored at the beginning of March to use OneCLI for credential handling. OneCLI is a lightweight proxy that sits outside the agent runtime, intercepts all agent requests to your control plane and then patches in the right credentials. This means the agent only ever sends…
There’s a pattern described here (prototype big, ship small) that’s valuable. It’s obvious as soon as you read it and you recognise the pattern in your own work. Like all great patterns I guess. The approach to finding “Small and Good Enough” models here is also sound. Again obvious once you’ve read it perhaps but its easy to be do the same work far less…
I have loved following along with Will Wilson’s Antithesis. Their blog posts and podcasts are consistently really strong. They’ve recently hired the creator of Hypothesis, one of those fuzz testing frameworks that I’ve read lots about but never used in anger, probably because of my poor understanding of their value (would the bugs I found actually matter?). It’s an…
Design language is definitely something I struggle with, I find prebuilt skills bundles a little heavy weight (not sure I want an extra 20 commands…) but I’ve bookmarked this to pick through later.
Folks have been having a lot of fun with autoresearch, I haven’t yet tried it out but I’m keen to. This article from the Langfuse team is a nice summary of the tradeoffs consciously (or unconsciously) being made: Autoresearch optimizes for exactly what you measure given the context you execute in. If your target function has gaps, it will find and exploit them. The community around…
Definitely starting to hear of more places rolling their own systems like this and having success. The feedback loop between the agent and the environment matters so much that commercial software likely won’t work in this space for a while, as the article points out: Current frontier models are very capable at a wide range of software engineering tasks, but they cannot synthesize a large…
Another nice idea from Intercom: So we built a publishing platform. In the same conversation where you create something, you say “share this” and your work - a fully interactive web app with charts, tables, and filters - gets published to a shared catalog with its own URL. It has versioning so you can iterate. Multiple people can contribute to the same page. View tracking so you know…
The truism in the title tells you all you need to know. I’ve been pondering this question lately (to review or not to review?) and I think this post lands a series of sensible points. Worth a read.
There were some wild stats out of Intercom the other week on their Claude authored PR ratio (they are through the 90% mark). In this twitter thread Brian breaks down some of the tooling they’ve built to make this possible. Rad.
Mileage might vary on how interesting the content is here, I think most of us are familiar with the arguments, but the microsite itself is gorgeous - the use of animations and different modes of interaction is so engaging that I had just had to link it here (mostly for my own bookmarks). I’m looking forward to more of the web looking and feeling like this. Enjoy.
I’m a sucker for a write up on coding agent architecture, I think because I enjoy learning about how the sandboxing works. Here’s one from Browser Use (who’s technical writing is often great). It’s a simple, effective design - probably exactly as you would do it if you started from a blank sheet of paper. I always wondered about the cold starts so learning a little about…
About a year ago at Nory we looked at the METR plots for coding performance, at that time the benchmark was predicting an 80% success rate for tasks that take humans about an hour to be hit at the end 2026. Opus 4.6 just went through that threshold (80% success, 1h 3 minutes task length, Feb ‘26 release) 8-10 months ahead of schedule. I remember thinking this time last year that the trend…
A slightly different benchmark to the popular swe-bench. Instead of using a handcrafted (and public) set of tasks, the team behind swe-rebench have built a pipeline that continuously gathers problems from public repos. Each model is also given the same harness. This means the problem set is less likely to have been subsumed into training data, and each model is assessed on the same level playing…
This effect has been very noticeable at Nory, folks really want to join and are attracted to the problem and the vibe. It was true for Bulb too. The best companies have it in bucketfuls. It gives you access to the best talent and this is a reinforcing positive feedback loop. What matters now is identity, community, belonging, and being out in the world, experiencing life in various ways. This…
I thought this was really a novel perspective in the oceans of content on gencode out there. It’s also an interesting idea, writing frameworks inspired by the past and then leaning into the model instinct on code naming and organisation. If the temperature of the LLM call is low and the context is stable then it makes sense that it’s guesses for naming etc. will be similar each time,…
It’s amazing how fast the zeitgeist is swinging around here: Every time you add a “smart” wrapper around model behavior - planning modules, verification layers, output parsers - you’re encoding what you think the model should do. But the model was trained on millions of examples. It has seen more patterns than you can anticipate. Your abstractions become constraints that…
Kyle Poyar’s writing at Growth Unhinged is normally solid and well researched, plus a handy source of benchmarks if you’re trying to evaluate startups, so it tends to be one I watch out for. This piece has some I treating stats on NRR for B2B SaaS/B2C SaaS and AI companies. The number of datapoints vary and we should take the results with a pinch of salt as they’re based on…
Very exciting to have an open source vision language model this capable. The queries described in the post are so varied (and work across different axis - semantic understanding, text understanding, object/spatial recognition), I think I this type of technology being cheaply/easily available is a big unlock for a lot of interesting product work.
Vicki Boykis’ year in review is excellent throughout (as her writing always is) but I loved this line in particular: The forking branches of a decision tree in a codebase, are likewise boundless, and the neat part is that there is no right answer. You are constrained by your business requirements, but the choice of implementation of those requirements is of an endless variety. It will depend…
Fun blog post from Andrew McCalip that attempts to build a model of the unit economics of orbital data centers. It looks like they’re just about feasible but really there’s only one player in town. This paragraph is key I think: This isn’t about talent. It’s about integration. If you have to buy launch, buy buses, buy power hardware, buy deployment, and pay margin at every…
Chad Fowlers take on principles for how the craft of software changes in the AI era: The metaphor I keep returning to is the phoenix: systems designed to burn and be reborn, continuously, without losing their identity. A regenerative system has a few defining traits: Clear, durable boundaries that outlive any implementation Tests and evaluations that define correctness independently of code…
A short talk from AI Engineer conference in which two Anthropic engineers (Barry Zhang and Mahesh Murag) make the case that you don’t need to build agents. Instead use a general purpose agent (Claude Code) and then write skills (skills are just pe-canned prompts expressed as markdown). The advantage of this approach being that skills are simple, versionable and composable. This last point…
Andrej nearly summarises where we are today: In this new programming paradigm then, the new most predictive feature to look at is verifiability. If a task/job is verifiable, then it is optimizable directly or via reinforcement learning, and a neural net can be trained to work extremely well. It’s about to what extent an AI can “practice” something. The environment has to be…
I can never really tell how useful economic analysis are. Often the fundamentals are staring you in the face and then the bull market runs for years. Timing is everything. However, I thought this was a solid summary that feels balanced and thorough, so worth sharing.
Tom McCarthy breaks out the current state of quantum computing. For me what’s valuable here is not predictions on potential commercial applications or the timeline but instead the heuristic to use to track progress and the clear line in the sand for a commercially viable technology: The key limitation is the size of the problem(s) that the QC can handle. Runtime, integration with real-time…
I enjoy posts like this deep dive from Joshua Rogers on “AI Security Engineers” as amidst so much noise they show the value that agents are adding at the frontier. Josh finds the tools generally useful, giving a good tear down in the post. I’m not quite convinced the tools are ready for prime time, there’s a few too many obvious gotchas outlined here (e.g. monorepo support,…
Interesting blog post on how database design changes for agentic workloads, first time I’ve seen the phrase “agentic speculation” used to describe agent querying patterns but it seems a good fit:
A very simple thing but this cheat sheet is great, even has simple diagrams for the different merge strategies built in, which is probably the most common area of debate (and confusion) when working with teams. Handy.
Nice deep dive on using LLMs for retrieval/recommendation. It’s a two parter, and there’s also a great guide to building a retrieval engine using a constrained decoding approach with vLLM and a HF hosted model. The whole thing is about 30 LoC.
I’ve really enjoyed following along with the Embrace the Red prompt injection series over the summer. Pretty much every major, hyped tool has been compromised by the same fatal flaw - LLMs today mix data and instructions in the same channel (the prompt) and the model doesn’t know how to separate the two things. The series finale (an old school self-replicating virus) is a particular…
Some words on AI strategy from Jamie: Perhaps, the AI-maxi strategy is building vertically integrated operating companies which wield strategic control to own the upside of AI, fundamentally transforming the economics of their business relative to incumbents, enabling them to counter-position and disrupt by selling directly to the end customer? e.g., Paloma Health, Convictional, Candidly Jamie…
Armin Ronacher (creator of Flask) has a great piece on agentic coding patterns, insightful throughout but largely centered on the uplift you get from effective tool use: Agentic coding’s inefficiency largely arises from inference cost and suboptimal tool usage. Let me reiterate: quick, clear tool responses are vital. For this reason, he tends to avoid MCP: The reason I barely use it is…
Elad Gil on AI Rollups: “It just seems so obvious,” said Gil over a Zoom call earlier this week. “This type of generative AI is very good at understanding language, manipulating language, manipulating text, producing text. And that’s audio, that’s video, that includes coding, sales outreach, and different back-office processes.” If you can “effectively transform some of those repetitive tasks into…
Shunyu Yao, a researcher from OpenAI who worked on Deep Research, makes the case for fundamentally altering our approach to benchmarking now we’re in “the second half”: Inertia is natural, but here is the problem. AI has beat world champions at chess and Go, surpassed most humans on SAT and bar exams, and reached gold medal level on IOI and IMO. But the world hasn’t changed much,…
Simon’s blog is a gold mine. He just runs that bit further than everyone else and it shows time and again. Here he uses Cursor’s GDPR subprocessor disclosure to document their stack (the use of Fireworks and Turbopuffer is the interesting bit here). The killer bit is the disclosure at the end though: When operating in privacy mode - which they say is enabled by 50% of their users -…
I think this is a very good post. Taking the time to test for yourself and understand how each model generation is useful to you, in your context is clearly going to be a big advantage. So much of the assessment of LLMs is vibes based that your own vibes matter most, so spending some time defining what they are is important. This blog offers a framework, and examples, of how to do just that.
Nice explainer that sets out the boundaries of the RL techniques now dominating progress in AI. The list quoted here neatly describes what the jagged edge of AI will look like for the next little while: Reinforcement learning is a powerful tool. Right now, though, it’s best used when: You have a verifiable problem: math, coding, robot grasping You have a way to generate a ton of data in this…
YC’s call for startups for the summer ‘25 batch includes a section on Fullstack AI, I’ve written about AI Rollups a few times on this blog, but it looks like the model might now accelerate. Coincidentally the same day OffDeal (a YC company) has published their blueprint for a rollup that takes on investment bank M&A. Somewhat unusually, there’s tonnes of detail in this…
This has been written about in a few places so I’ll keep it brief. It was interesting that one of the root causes (note, not the sole cause) of the ChatGPT sycophancy issues was the feedback loop from the thumbs up/down data on posts, from their blog post: “We also teach our models how to apply these principles by incorporating user signals like thumbs-up / thumbs-down feedback on…
Interesting paper from Cohere, I think this might cause a bit of a storm - basically it’s an investigation into biases towards closed source model companies (OpenAI, Meta, Google DeepMind are named) in Chatbot Arena. There’s three ways that the proprietary shops are favoured: There’s private testing practices that means these model providers are able to test multiple variants…
Insightful post from J Betker on the MoE architecture. Here’s a few grabs: The fact that MoE has great scaling properties indicates that something deeper is amiss with this architectural construct. This turns out to be sparsity itself – it is a new free parameter to the scaling laws for which sparsity=1 is suboptimal. Put another way – Chinchilla scaling laws focus on the relationship…
This is really good, well worth the investment of your time. There is a lot of novel insight here that will shortly become de rigueur. There’s a few bits worth calling out. The models are now heavily tuned for too use as we all know. gh cli use is baked in: Claude knows how to use the gh CLI to interact with GitHub for creating issues, opening pull requests, reading comments, and more.
Another AI prediction, but I think this one pinpoints some of the blockers much more clearly. In summary: Roughly: generalist scaling does not work or, at least, not well enough to make meaningul sense for material deployment. Instead, most development, including agentification, happens in the smaller size range with specialized, opinionated training. Any actual “general intelligence”…
Well if Kent Beck is doing it: Been vibe coding like a fiend. Task breakdown is a highly leveraged human decision. Coding models are both non-deterministic & sensitive to initial conditions. You’ll get very different results having your agent implement Task1->Task2->Task3 or Task2->Task3->Task1. I don’t have good heuristics yet, I just observe that when I try to implement “the…