A trip report into agentic coding

The long-and-boring disclaimer bit: if there’s a spectrum of AI enthusiasm, I’d put myself on the lukewarm end. There are people I know who really hate AI, some who really like it, and some who generate lots of pelicans on bicycles.

One of the downsides of being involved with technology is you have to listen to lots of interesting but deeply wrong ideas. Some of my favourites of the last twenty or so years include:

• “the WS-* dream will be achieved once we’ve written 30 more unreadable specs”
• “XML schemas will magically fix all data integration issues”
• “Agile® means software won’t be crap anymore (and if it doesn’t work you’re probably doing Agile® wrong)”
• “an enterprise blockchain will magically fix all data integration issues”
• “let’s reimplement (a vague understanding of) contract law in JavaScript Solidity Rust”
• “large language models will magically fix all data integration issues”

Stories like these exist mostly to convince people to buy shiny expensive computer things. Said shiny expensive things have a habit of breaking down in catastrophic ways on your network which then makes it your problem. This tends to happen about 3am on a bank holiday weekend, often when the last thing you want to be doing is looking at logs in the AWS console. This induces a healthy distrust of anything smelling vaguely like hype, a strong preference for “boring software” (three cheers for Postgres) and deep suspicion whenever powerful non-technical people (former prime ministers, High Court judges, the head of sales) become too concerned with the particular brands of technological sticky tape the technical people trust enough to grab out of the toolbox first.

But this particular shiny thing has continued to remain very shiny and economically/politically interesting, and there is some level of professional duty to explore and understand shiny new things, to see if there is a thing there.

My first (and very non-negotiable) principle in these explorations has been “I don’t want to give anyone money”. Of all the things I could possibly do with money, giving it to anyone in Silicon Valley in return for algorithmically-generated Python code is very far down the list.

(There’s a broader reason for this principle too: I think programming—and being able to use your computer the way you want to—ought to be a thing anyone can do. To a fully committed vibecoding enthusiast, a $200/month subscription may be a bargain that, if the promises are all true, gives a large return on investment. For a middle class professional on an average American, British or EU income, that may be affordable. But $200—or even $20—is a much more significant proportion of the disposable income of someone in Kenya, Mexico, Nigeria, or the Philippines. I’m interested in technologies that are for everyone.)

My second principle has been “I’d prefer to use open weight models, preferably on my own machine”. As I don’t have £6,000 lying around for a top spec Mac Studio, I’m using a 16GB MacBook Pro. This does not, apparently, give much scope for running the chonkiest of models, so I’ve bounced back and forth between local models and the free models provided by Google and Groq.

The models I’ve mostly been using then:

• Local: qwen2.5-coder:7b (Ollama)
• Groq: llama-3.1-8b-instant and llama-3.3-70b-versatile
• Google: gemini-2.5-flash

If I were doing this on code that mattered, I would not be using the free version of Gemini because Google retains your prompts “to improve our products”. No thanks.

My final principle: “the agentic harness/coding tool/etc. ideally needs to be open source and model/provider agnostic, and not rely on any specific text editor”. Every time I’ve used a closed source or proprietary development tool and ended up relying on it in a non-trivial way, it has been a big mistake. Giving up control of your tools is a dead end. One of the redeeming aspects of software development is getting to use tools like Emacs that you can shape entirely to your own design (and which cost nothing) rather than, say, paying Adobe £77.33+VAT/month for the privilege of using a suite of applications that seem to get worse every few months.

Quite a while back, I briefly tried Gemini CLI using the free Gemini plan, and also Claude Code (using some non-Anthropic model). I can’t quite remember for what but it took a long time to not do very much. I’ve tried out Picocode, and OpenCode, but mostly settled on Aider.

I tried Picocode with an Ollama model a while back. It didn’t work out that the model I tried it with did not have the relevant support for tool calling, so when I asked it to generate a simple Hello World script, the harness just kept spamming the model over and over generating all sorts of fun varieties of Hello Worlds, moving from Bash through Python to Rust then back to Bash, but not being able to compile or run any of them. Kinda felt like a robot screaming from the void; I’m very glad said experiment was just wasting the CPU cycles of my own laptop rather than API credits.

OpenCode fits in the trend of incredibly pretty TUIs (thank/blame Rich, Ratatui, Textualize, Ink, and also the spread of modern terminal emulators like alacritty and kitty in place of boring old ones), but out-of-the-box it uses lots of tokens.

Aider, by contrast, doesn’t have all the agentic stuff in there like OpenCode/Claude Code does. Which is handy as the models I’m using (other than Gemini) don’t have particularly large context windows. When I tried OpenCode, the Groq models just go “nope, too much”. On a very small project. Perhaps I’m using it wrong.

One other observation I have about the marketplace for these agentic frameworks/coding tools is surprise at the amount of weird premature performance drama there is in the comments on places like Reddit. People find it oddly important whether it is built in Rust or Python or NodeJS or whatever. They’re I/O bound. They’re ultimately reading a bunch of bytes from STDIN and disk, sending them off to a model endpoint (whether local or elsewhere) and parsing the result, then shelling out to git and whatnot. Compared to the multi-gigabyte model that the agent is talking to, the overhead imposed by a few Python object allocations isn’t going to matter. But people need something to argue about, so I guess we get a really dull version of the console war in this domain.

Interlude: installation woes

I’ll detour slightly and talk about Aider’s installation process. It’s Python, and Python packaging is a legacy of sadness that’s mostly been fixed.

My first port of call was to use the Nix package:

$ nix-shell -p aider-chat

Nope. Doesn’t build on macOS. Maybe I could ask a robot why, but whatever. I’ll reluctantly install it in ye olde world way.

The starting point in the documentation is:

$ python -m pip install aider-install
$ aider-install

This installs its own Python, because one cannot have enough versions of Python lying around on one’s laptop. And there’s a curl | sh option which I immediately noped out of. I want it installed in a known place that I can nuke if it causes problems, and a random Python install script and/or curl | sh aren’t quite to my taste.

If you’re doing Python in a modern way, you’re probably using uv so you can then use uv tool aka uvx which is much more reasonable. The documentation says:

$ uv tool install --force --python python3.12 --with pip aider-chat@latest

I don’t see why in particular it can’t run on 3.13, and after consulting a GitHub issue about version compatibilities I went with:

$ uv tool install --force --with audioop-lts aider-chat[browser]

(The [browser] means that you can boot up a web view with --browser.)

The Aider documentation is… okay. I’m kinda curious whether they vibe-docced it or not, because there’s bits of the documentation I think could do with some significant improvement.

Let’s build something

My main test then was a simple one: build a very tiny Flask app that does currency conversions. It’s a nice developer interview type of problem. If people are building entire companies or whatever the LinkedIn hype is, surely I can get a nice little webapp that lets me put in pounds and gets dollars without much trouble.

I needed some sample data, and I didn’t particularly want to sign up for an API key. Fortunately, I found a GitHub repository that uses GitHub Actions to get some currency conversion data from somewhere and pushes them in JSON format onto cdn.jsdeliver.net. I grabbed the latest one as JSON with USD as a base currency.

First problem: it contains a bunch of data for things other than proper fiat currencies, including cryptotokens and commodities. I don’t care about Solanas or Moneros or ounces of palladium. Nor do I particularly care about pre-Euro currencies like francs, escudos or Deutschmarks. Gotta filter those out. Handily, I’ve got this large language model right here. Surely, in the training data, there’s enough material to be able to filter out current, government-issued currencies that an ordinary person might use on their holidays (like dollars and yen) from dog-themed memecoins, precious metals, and other crap I don’t care about.

I ask Aider to create a copy of the file with the extraneous stuff filtered out. It chunts and whirs and gives up. I switch to a different model. It chunts and whirs some more, starts emitting JSON but then gets stuck in an infinite loop printing out the USD to Chilean peso conversion rate over-and-over until I hit Ctrl-C. I contemplate seeing if I could add a custom tool or an MCP server or something so it could call some kind of API that would answer the important question of “is this thing a real currency or some crypto nonsense?” for each currency… and then gave up because the yak shaving budget was exceeded. I opened up the JSON file, and edited to remove all the entries I care about, namely USD, NZD, GBP and EUR.

The LLM generates a REST endpoint that looks right. But there’s some problems.

Firstly, it checks to see that both the from and to currency are in the currency conversion dict and returns a 400 error if they aren’t. Reasonable. Except it doesn’t actually do that. The currency conversion table is a dictionary of currency symbols to base (USD) conversion rates (a dict[str, float] in the parlance of typechecked Python). It does this:

if from_currency not in conversion_table or to_currency not in conversion_table:
    return jsonify({"error": "Invalid currency"}), 400

That won’t check against the keys. Easy enough, I just add .keys() to both of the subexpressions and we’re good.

Next I see another problem: if I ask it to convert from GBP to EUR, it’s converting the GBP amount to USD, then the USD amount to EUR. In a real system, we’d potentially have different conversion rates between different pairs, so we’d probably want to check to see if we had a direct GBP to EUR conversion pair in our data, then fallback to converting through USD as a base if not.

There’s a more fundamental problem. The logic it gives me doesn’t work if I want to convert to or from US dollars because the base unit isn’t in the list of currencies.

There’s a really simple fix for this: I open the JSON file, add USD, and set it to 1.0. If it weren’t a toy example, that’d be a more significant logic issue I’d want to fix. But I felt that explaining the problem in English to an LLM was going to be a lot slower and more frustrating than just fixing the problem. If you don’t know how to code, it’s magic. If you do know how to code, just changing the code is often preferable to writing an English description of how you want the code changed.

Next, I have the assistant generate some tests. We’re going to need a pyproject.toml and some dependencies - Flask, for one, but also pytest.

I ask Aider to generate this kind of project scaffold. It does it the old school way: requirements.txt and tells me to use python -m venv and pip. Been years since I did that. Proper nostalgia.

Time to prompt it again. No, I don’t want a requirements.txt file, I want a pyproject.toml and I want to use modern tooling (uv would be nice, but I’d settle for poetry). Even more old school nostalgia…

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

Why setuptools? This is a greenfield project, and I just said I want to use uv. (Also, it’d be nice to have a requires-python field, and a .python-version file, and an override for tool.pytest.ini_options.pythonpath, but I’m perhaps being too picky.)

In the dependencies Flask 2.3.2 (released 1 May 2023) gets specified not 3.1.3, along with pytest 7.4.0 (released 23 June 2024) instead of 9.0.3. (uv locks current versions, thankfully.)

pytest ends up in optional-dependencies.dev rather than dependency-groups.dev. That’s not right. I moved it by hand, because if you’ve got Vim stuff burned into your brain, tapping Y 3k p is a lot more efficient than “please robot man, move the pytest dependency to dependency-groups” and waiting for an LLM to chunter for a bit.

The test code is rather clever in mocking out the open() call that reads the currency JSON data in, and returns some static test data. Much as I’m not that keen on AI, I’m even less keen on the tiresome task of writing mocks and stubs, so that might potentially be a win.

Except, no. The unit test code generated mocks the implementation the LLM thinks the code has, not the actual implementation the LLM itself just produced. Bravissimo!

I realise I can just read the currency data into app.config['currency_data'] and then overwrite that in the fixture that generates the test client, thus avoiding any need to mock the open() call at all. Refactored. Unit tests are green. Happy days.

Next I ask the agent to generate me a really simple UI.

And it does. The UI looks basic but usable. One problem: it doesn’t work.

The data has the currency codes stored in lower case. The unit tests pass because the mock data the LLM generated has them in upper case.¹ The form in the UI has them in upper case. If only the underlying data were available to the agent to look at when writing the tests. (Presumably “USD”, “GBP” etc. appears more in the training data than their lower-case counterparts.)

I add two .lower() calls in the code and we’re good.

There are no obvious howlers in the HTML, CSS or JavaScript. It is modern JS using the Fetch API (and templated strings and await and a few other niceties), does some basic DOM manipulation and doesn’t rely on any third party libraries, and I can understand it in a way I can’t with the Tower of Babel of gulpified-reactive-left-padded-flux.vite.jsx or whatever. Passing grade.

About the only complaint I can find on the front-end is the font-family stack: Arial then sans-serif, with none of the much nicer sans serif fonts you could put higher on the stack. Giant floating point matrices can perhaps be forgiven for having bad taste in typography, but, still, eww.

Vibe check

I can see some advantages with this way of working, specifically around rapid prototyping. But every stage in the process returned code with problems. Sometimes there were trivial fixes to those problems, but sometimes the very people who the tool ought to be of most benefit to (newbies, language switchers etc.) would probably be the most puzzled by some of the oddities.

“Are we using encoding these values as lower or upper case strings, and is there a way to normalize those values?” is a fairly basic thing most reasonably competent programmers should be able to fix, and the knowledge and practice of how to do so is transferable between languages. I’m hoping that if you are programmer who has never written Python before, you can probably work out what str.lower() and str.upper() do.

I was reasonably impressed with the frontend. It’s probably not something you’d actually put into production, for a whole bunch of reasons. But it is a basic scaffold one could use for internal tools or demos or hackathons.

The problem I really had was with the weird, non-obvious stuff. If you’re not very familiar with Python, knowing what all the various bits of a pyproject.toml file are for is not exactly clear. That’s the kind of thing where an AI could actually help, and yet the bit that I spent way too much time fixing.

Getting that stuff set up right is a faff. There is a rather big gap between uv init and the Giant Ball of Stuff that normally goes with modern software development: dependency management (won’t someone please think of the supply chain attacks?), testing, type checking, CI, code coverage, linting, LSP/editor integration, containerisation, Makefiles/Justfiles, documentation generation, licensing, .env/secrets management, deployment, managing databases (migrations), caching, message queues, reverse proxies, mocking/stubbing of third party cloud resources, instrumentation, performance testing… the list grows ever longer.

(A lot of the reason that is all incredibly frustrating is it is the place where all the various seams all rub up against each other, and where they are hastily connected with some lowest-common-denominator lingua franca like Makefiles and YAML. All of project configuration and management is basically a hand-me-down patchwork quilt that we keep hoping will one day miraculously transform into a perfectly fitting Savile Row suit.)

There is some plausible advantage in the new project scaffolding, though you can get a fair chunk of the “getting a new project started very quickly” benefit through opinionated and shared project templates (in Python-world, tools like cookiecutter and copier), or with more radical approaches like Devenv (other brands of Nix/Guix-style declarative reproducible magic are probably available).

Whether you have a project template lovingly crafted by an old wisebeard or an LLM doing the initial generation of the Giant Ball of Stuff doesn’t really do much to fix the layers of crufty complexity. When you need to change it, you still need to untangle the damn thing and know all the Makefile footguns, why the Norwegian language suddenly broke CI, or (heaven help you) something truly cursed like m4 macros.

At the end of this very unscientific experiment, what do I reckon? I don’t love it. I don’t hate it. It’s almost too pedestrian to have really strong opinions about. I’ll probably play about with it some more in the future, mostly because it is kinda fun to experiment with—particularly without any mangerial coercion. But I still suspect Fred Brooks won’t have to posthumously amend No Silver Bullet for a while yet.

I’ll end with a disclaimer (reading comprehension in the era of TikTok is hard): this is merely a recounting of my own experience trying these particular tools out. No doubt, I’m certain I’ve probably used the wrong models, language, tools, and/or prompts. And probably on the wrong computer, on the wrong day, while sitting in the wrong position, or while wearing the wrong type of socks. I may also be an idiot, have voted for the wrong person, or perhaps I broke everything through egregious wrongthink about our future robot overlords. Maybe if I’d been born a Sagittarius or committed fewer sins in a past life, it’d all have worked out differently.