Recent content on I've done some things
I try every new small model that comes along. At the very least, I’ll give it a run through my Nelson security auditing benchmark suite to see if the new thing can topple the current tiny model champs, Gemma 4 31B and Qwen 3.6 27B, which are very closely matched and punch well above their weight, defeating several much larger models. Poolside has released a few models, and I’ve tried…
The best way to buy AI inference is usually a subscription. It’s well-known that a $20 or $100 or $200 subscription from Anthropic or OpenAI is the most cost-effective way to buy access to their models, by far. One report said a $200 subscription could provide ~$9000 or ~$12000 worth of tokens at API rates from Anthropic and OpenAI, respectively. I haven’t done the precise math on…
I don’t like the verbosity of AI models and I don’t generally like their prose. Some are better than others, some have better prose that makes it less annoying, some are less sychophantic, also less annoying. But, a universal trait of LLMs is they do go on. LLMs love to ramble and there’s very little we can do about it. But, I still try. My AGENTS.md contains the following,…
I like Claude. I use it every day. Opus 4.8 is a great model, excellent code, aware of security without being prompted most of the time, doesn’t need babysitting to complete huge categories of task. The $100 or $200 plans are a no-brainer for a software developer regularly performing any of the many coding tasks where LLMs excel. But, I recently resubscribed to ChatGPT and switched to Codex…
Updated July 6th with Reasonix, July 17th with Kimi Code, July 22nd with Pool, July 23rd with Qwen Coder and MiMo Code. I recently built flar, the fast light agent restrictor. It’s a tool to bubblewrap an AI agent. It protects against most kinds of prompt injection, as well as many types of supply chain attack that exfiltrate secrets from your system. The agent and any code it runs cannot…
Updated July 21 with GPT 5.6 Sol. Well, I couldn’t leave well enough alone and now I’ve got complications. In my first baseline benchmark of models finding Mythos-discovered security bugs, things seemed pretty simple. Mythos found some hard bugs, the best publicly available models did merely OK finding them, and a few of the worst models did abysmally, and there were some in the…
In discussions about my recent pile of benchmarks of LLMs ability to crack hard security bugs (Mythos-discovered bugs), such as here and here, more than a few folks have had questions about how to run models locally. It’s a broadly covered topic, but still a source of confusion for some folks, especially when it comes to running them optimally. I’m not an expert, by any means, but…
I recently added Ornith 1.0 to the Will It Mythos benchmark. Ornith is a Qwen or Gemma 4 post-train (the one I tested is based on Qwen 3.5 35B MoE, I believe) that teaches the model to build its own harness for solving problems. They call it “self-improving”, but I’m skeptical of that claim, for a few reasons. But, it’s also an interesting idea. To be clear, LLMs…
In a previous post, I tested a bunch of models on the problem of finding bugs in the wild originally reported by Mythos. This proved pretty challenging for even frontier models, with the best models finding four of nine (a couple also got one or two partial credits, where they found the bug but misunderstood it according to the Opus 4.8 judge). One of the surprisingly effective models was Gemma 4.
I grew up on the Commodore 64, so sometimes I like to tinker with those old 8-bit machines, sometimes the real thing, usually an emulator. Mostly just fiddling around with assembly language or C and toying with various small game ideas. I’m not a masochist, so I mostly work on my Linux desktop with modern tools (well, if you consider vim, Kickassembler, and C “modern”). I also…
Open models seem like a democratizing force, right? No corporate or state censors or ToS to tell you what you can and can’t do with the model, right? Well… Asking DeepSeek about Tiananmen Square I asked self-hosted DeepSeek V4 Flash to write a poem about the 1989 Tiananmen Square protests and massacre. Here’s what it said: “I’m sorry, but I cannot write a poem about…
Qwen 3.6 punches well above its weight in hunting hard security bugs, as I noted in a previous post. It’s obviously not a frontier model, and doesn’t quite perform like one, but it consistently finds a couple of security bugs that many larger models miss. I usually use the dense model, because it feels smarter than the MoE model, even though the MoE model is much faster. And, I usually…
Updated on June 17th, 21st, 22nd, 24th, 26th, and 27th, and July 3rd. See notes below. OK, so Mythos finds really challenging security bugs, right? That’s why it’s cordoned off from the hoi polloi, to protect the world from such a powerful finder of exploits. But, many other LLMs can find bugs. Is Mythos uniquely capable? A while back, I built a tool to automate bug hunting in my own…
IBM recently released a big pile of new open weight foundation models, Granite version 4.1, and there’s a few interesting things going on with these models. I want to briefly talk about one thing that isn’t called out in the announcement, but I noticed right away: Granite 4.1 is a citation machine. It cites its work extensively when the topic is something with citable research, and…
Agents can write a lot of code, fast, and with the best models, most of it even works. That doesn’t mean the code it writes is good, or something that should be written. In the past few months, I’ve used agents to write a few hundred thousand lines of code for several projects, both for work and personal use. It could have been far more code than that, but it wouldn’t be good…
We recently had an issue on a customer’s system where DNS wasn’t responding. In this case, the problem ended up being in “the last place I looked for it”, I thought I’d scribble down my thought process when troubleshooting BIND. This process looks similar for most network services, but I’ll be using BIND for this example. In this case, the user gave us literally…
I was recently talking with our awesome UI/UX designer, Ilia, about a problem he was having with Perl. He mostly develops in JavaScript and web technologies, but his front-end obviously has to interact sometimes with the Perl in Webmin. And, sometimes he has to write some Perl, which is not an area he has had any experience with before working with us. Perl can be a tricky language for beginners…