Last weekend, I dusted off a Scheme implementation (Arete) I wrote in 2017-18 in C++ and itself. It was a fun and educational project but eventually dropped off with a long TODO list.

While there’s a lot I would like to do given unlimited time, this go around I just wanted to see if I could make it faster using the ideas from autoresearch and put GPT 5.4 and Opus 4.7 through their paces.

Autoresearch is a repository from Andrej Karpathy containing a workflow he used to optimize training of a small ML model, later adapted by the CEO of Shopify to find optimizations in Liquid.

Using autoresearch and the new models I was able to speed up the implementation by 25-35% on general purpose benchmarks.

#Autoresearching performance improvements

I started from Karpathy’s prompt, adapted for Scheme instead of training, and used both Codex and Claude Code at various points. Here’s my prompt as of today and the changes landed over the last two weekends.

#Setting up a harness

Initially I just started by having agents read the prompt and then go. A few adaptations I made over time:

• I used Ralph Loop to prevent the autoresearch session from halting. Prompting alone (as in Karpathy’s prompt file) wasn’t enough, and work would stop after a few experiments.
• I ran each experiment in a sub-agent to keep the context window clearer, and ran them serially: since I’m only on the $100 plan for each, even this soaked up my limits on both, and it made monitoring and merging easier. Parallel experiments should be workable.
• I spent time on the benchmarking subsystem itself, setting it up to record and compare against a baseline easily, such that the test would look something like this:

python3 utils/benchmark-report.py compare --baseline scratch/compiler-baseline-2026-04-18-peval.json

Earlier runs with a more primitive benchmarking system (e.g. needing to chain multiple long-running commands and compare text) seemed to go more poorly.

#The importance of a good benchmark

By far the most important thing is having a very good benchmark or set of benchmarks to use as the north star for the autoresearch session. Language benchmarks are a notoriously tricky (and touchy) subject.

My initial approach relied on having the model select a set of benchmarks to track over a session. I found this led to overfitting optimizations, or selecting benchmarks that would have been impossible to measurably improve by working on the subsystem I selected for a given session.

#How it went

Once I’d worked out the kinks with the benchmarking and harness, it found some pretty interesting and simple optimizations. Some of these were in retrospect quite obvious.

For example, (null? val) in Scheme checks whether a value is an empty list. In Arete this is a simple arithmetic check, but it was done by calling out to a separate C++ function to do it even though it could have been expressed in the virtual machine itself. Here’s the simple fix.

#Adapting the codebase to autoresearch

The nanopass framework is another interesting idea from the Scheme community. Nanopass is simply the idea of writing many small passes which transform the code incrementally instead of large, complex compiler passes which try to do a lot of work at once.

I think a rewrite in this direction might be really beneficial for autoresearch. It’d be easier to drop a self-contained pass on the filesystem and test it with a config change, rather than re-processing and trying to modify increasingly large monolithic files.

#Not autoresearch: writing a native virtual machine

Ultimately some of the bigger performance wins came out of using the new models to land some bigger projects that were on my mind outside of the autoresearch session. For example, LuaJIT has a virtual machine written in native code and is quite fast even with its legendary JIT disabled (Mike Pall explains here).

I was able to use Opus 4.7 to port Arete’s C++ virtual machine to native AMD64 in a similar approach, which resulted in a 10% or so speed gain.

#Being ambitious and hands-off at the same time

The codebase itself seemed to constrict the possibilities the agents would consider. “Write a virtual machine in native AMD64 code” wouldn’t have been attempted had I not pulled it out as a separate project, and other major refactors (like going from a stack to a register-based VM) never came up either.

Maybe a prompt/harness issue, maybe just a patience issue on my part. A longer run might have surfaced more radical optimizations.

#Conclusion

This was a fun hack over two weekends that let me play with a few different ideas I was curious about but haven’t had the opportunity to try. I remain perpetually surprised at how much agents are improving; I don’t think any of this would have been remotely possible a year ago, maybe even 6 months ago.