I can program and like games
Eight months ago I was producing roughly 90% human written code and 10% AI written code. This has switched surprisingly rapidly and now my code is probably 90% AI. So what do I do all day? I’ll go through a change that I made to a matrix-multiply kernel, for which I’ll sadly have to be […]
The book Babel, or The Necessity of Violence: An Arcane History of the Oxford Translators’ Revolution is fascinating. And I’m not sure how much of this the author intended. It’s mainly fascinating because it’s trying to discuss colonialism, technological progress, and inequality with characters who are all stuck in the 19th century and are unable […]
It started off a few months ago, when my iPad suddenly couldn’t play videos any more that it had recorded. It would show the first frame, but hitting the play button wouldn’t do anything. I googled around but couldn’t find anything. I had recently installed an update, so I figured I’ll just wait for the […]
I made a website to explore FRED data broken down by US president and party. This is obviously motivated by the current president. During the last election I was frustrated by how many nonsense arguments there were being made. Like people voting for republicans because they were hoping for a good economy. This seemed exactly […]
The US has been acting powerful recently and it reminded me of this question: What does it feel like to fight against a powerful AI? Not for normal people for whom there’s no difference between competing against a strong human or a strong AI, (you lose hard either way) but for the world’s best humans. […]
I really like Semantle because I noticed that progress is similar to the description in Why Greatness Cannot Be Planned: The Myth of the Objective by Kenneth Stanley. The objective function is very clear and you can hill-climb your way to the top, but hill-climbing is actually very difficult in high-dimensional spaces, so you need […]
I submitted a tiny patch to flash attention. The necessary typing for the change takes less ten seconds, but the overall change took more than ten hours So where does the time go? It started when coworker had a bug where cudnn attention would crash randomly. We looked at his unreleased changes and concluded that […]
If you look at the source code of a modern open source LLM, it looks very similar to the transformer described in the “Attention is all you need” paper from 2017. It’s just a stack of exactly three components: attention blocks, matmuls, and norm layers. The big algorithmic changes, like Mamba 2 or linear attention […]
Daylight savings just ended so this post is coming exactly too late. But I was talking to a friend about how we just adjust 15 minutes per day, which is quite easy. She was saying “oh I wish I had that kind of foresight.” But we were talking Friday evening, and daylight savings didn’t end […]
I’ve wanted to write about this since the studio closed a year ago. Now that Contraband is also canceled I think it’s time, especially since Contraband was one of the big reasons why I left the company. The blog post turned out much bigger than expected though. There was a lot to get off my […]