Experiments with agentic coding [01KC]
On the side of Pterodactyl, I’ve been experimenting with agentic coding via Amp, who have kindly given me $10/day in tokens. I realise this sounds idiotic to some of my audience, but it is actually very good at doing things like “Find where in my entire codebase this off-by-one mistake is coming from”, or “Figure out how to use the API from this undocumented library to overlay the active buffer states onto the CAS database ingested from disk”, etc. It’s also great at dealing with TypeScript and Visual Studio Code extension nonsense.
Prior to trying Amp, my experience with LLMs had been limited to chatbots. Those experiences were very frustrating. From my experiments, I’ve concluded that the “agentic” approach is a few orders of magnitude more effective. One thing that pays off is to have a testable loop that the agent can refine, but even in cases where you are doing things that are barely testable or observable (like GUI programming with 40-year-old frameworks), simply having the entire codebase in the context seems to be a huge improvement. I deal with far fewer hallucinations than with the chatbots, where the interaction was 95% hallucination.
Obviously I carefully review every piece of code that the thing writes and change it myself. I treat it as a learning process for me, because I tend to learn best by example. I’m not vibecoding; I’m simply using an extremely powerful and untrusted example-generation and testing model. If anyone is interested, I do recommend reading Mitchell Hashimoto’s post on the topic, which I found very enlightening. I don’t think I will go as far as he did for a number of reasons (including cost!), but following his work has been eye-opening.
It is difficult for me to talk about this because I don’t want to contribute to unjustified hype. On the other hand, I’m getting a lot of utility from it. I do think it is this useful to me only because I am an expert software engineer with many years of experience. I worry about my students, who will not have that context; already, I see signs of what I would consider LLM-induced brainrot in some of my students. We will have to figure out a realistic model with which to confront that brainrot.