A few months’ experimentation with agents [01KO]
For a while the idea of using LLM-based agents had been merely hypothetical (because I have no money and live in one of the most expensive cities in the UK), but I managed to get $10/day in free tokens from Amp which has made it possible for me to do some experiments. I also got some free tokens from GitHub, which allows me to experiment with other models including Claude’s Sonnet.
Let me say before I comment further that I hear all the critiques of LLM companies and their actions, and I agree with these critiques. I will not be giving these people any of my money, even if I had money to give them (which I don’t).
I have not attempted to do much “easy” stuff with agents because there is no point: I find this very wasteful, because the cost in tokens is very high to do things that take me very little brain-power. But I was very interested in whether agents can help with difficult things, like the typical needle-in-haystack bugs that one finds in an experimental state-of-the-art implementation of a proof assistant. My experience here has been eye-opening; I will summarise the main takeaways:
- Claude’s Opus model is incredibly costly to use, but it actually is able to find and fix these needle-in-haystack bugs. Not all of them, of course. I would estimate that I have saved on the order of a couple dozen debugging hours in the past month via Opus. However, finding and fixing a single challenging bug with Opus will typically cost in the range of $6-$10. (Less challenging bugs are cheaper to fix, but I don’t need automated help for those!)
- Claude’s Sonnet model is essentially useless for any kind of sensitive or challenging work; it is probably impressive to people who have already succumbed to Claude Dementia. I understand you can get it to churn out a lot more code at lower cost, but churning out large amounts of code is not useful to me (and I don’t think it is useful in the long term to anyone). It is very prone to confabulation, hallucation, and spinning in circles of “Aha! I found it! Hmm, let me look into this deeper; ok I didn’t find it; Aha! I found it !”, and then ultimately masking the bug with some idiotic “fix” that makes the test pass but doesn’t solve anything.
- Qwen’s coder model is interesting, but not very good. Again, it can probably be used to churn out lots of code, but not for sensitive tasks.
My main takeaway from this experience is that a state-of-the-art model can be very useful as a rubber-duck or as a way to find deeply challenging flaws in my code. But the cost is extremely high and the token economy is unbelievably bad. Even if it weren’t for the ethical issues in play, if I had to pay for these things in order to use them, I would not pay; although you can apparently build lots of React shit-apps for $200/month, it would cost me closer to $1000/month to do something at the level of difficulty that I actually consider worth my time, and this is an amount of money that I will never have to spare (and if I did, I would put it to a better human-oriented use).