Blog Posts

My Current Research Direction

Here's an attempt to explain my current research direction. A manifesto, some thoughts and experiment ideas, and a roadmap of sorts.

How to Write a Compiler Without Going Insane

Thoughts About LLMs, Data, and Optimization

First you select a metric. You acquire suitable data. Then you apply optimization pressure. Easy, right?

The Competition For Contributors

The Craziest Bug I Have Ever Witnessed

I thought I had seen it all. Nah. Not even close. Julia has a feature where you can report bugs and crashes from the command line. You just run your program with julia --bug-report=rr, and it sends a crash report plus an rr trace of the process to our servers for us to take a look at. Sometimes we get some crazy stuff. This is in fact not the first time that someone from the Julia team has found…

Scaling Agents to Unfathomable Depth and Context With Zeroth-Order Optimization

An interesting research direction I'm thinking about. Posts like this are often ignored because they are not directly applicable to anyone's work. But this might be a new scaling paradigm. So I've sort of dropped everything to work on it, study it, understand it. This is a writeup of some things I've learned.

Hyperparameter Heuristics

Suppose you've invented a new neural architecture, or optimizer, or whatever, and you want to scale it up. You have 100 million dollars to do this with, and you want to train the best model you possibly can with that budget. But how should you choose your hyperparameters? There's no existing literature on your methods! Who knows what settings are optimal, and if you guess wrong, you're out a cool…

LLM Code Optimization

It's long been assumed that for most programmers writing hardware-optimized code is too hard. Few know, or want to know, how to exploit memory hierarchies, NUMA nodes, vector instructions, stop registers from spilling, or exploit the full memory bandwidth of the CPU. This is tragic. In the age of LLMs and tooling, we should expect more from our software and our hardware.

Descending Into the Stack and Madness

AS-Safe glibc stack traces. A descent into madness. A death by a thousand cuts. Call me melodramatic, but all I want is a nice, pretty, formatted backtrace from a signal handler. What am I to do?

Cursed Code Collection

I have a pile of interesting programs, and nowhere to put them. So I'm putting them here. The shebang for C is just a little bit longer than one line.

Thoughts About Entropy and Objectives

A bunch of Reinforcement Learning Pre-Training (RLPT) papers have come out recently. They're cool. They even work kinda mostly. They can improve benchmark scores. So, like, probably there's gotta be something there. It's doing something. Unfortunately, nobody seems to be in agreement on what they are doing or why. Here's my understanding of the problem, and a take on a path forward.

Prompting LLMs and Humans

Often you ask questions because you want to hear the other party's opinion. There are other reasons you may purposefully ask a leading question, but let's focus on the "soliciting opinions" case. Human or AI, you seek their perspective because it has value to you. You genuinely want to know. But when you phrase a question, you shape the answer you get. Every word matters. The context you provide,…

Grifters

Forgive me, but today we will be talking about politics. I will be speaking in hyperbole. Note that for every broad generalization that I make there is bound to be a counterexample, and for every simplification an elaboration. But this is, for the most part, what I believe. I am open to argument in good faith.

A Treatise On ML Data Infrastructure

If you want to do machine learning, you need a lot of data. If you need a lot of data, someone must collect and process a lot of data. If someone must collect and process a lot of data, they need infrastructure. Here is a way to build that infrastructure.

Safety and Correctness

Software safety is a lofty, but poorly defined goal. Everybody has their own opinion about how to achieve "safety" in software. However, nobody really seems to be able to define what that means.