Thinking Thoughts
The recent publication of Recursive Language Models from MIT has sparked debate in the AI community. Some call it a breakthrough. Others dismiss it as “just agents with extra steps.”
I started using aider last month as a way to get back in shape after a long leave from work. Eventually aider sped me up a lot, but I had to learn some new ways of approaching development. It was different from using cursor in that you don’t really see the code until it’s being edited. This doc is a compilation of the notes I took as I worked on a few projects, which I think will be helpful to…
This is a short memorial post for my dead Twitter summarizer project, aka “Twitter at a Glance.”
Some sentences in this blog are generated by GPT-3 Davinci. In those cases (and some others) words are highlighted according to how probable GPT-3 thinks they are.
When I first read Google’s RETRO paper, I was skeptical. Sure, RETRO models are 25x smaller than the competition, supposedly leading to HUGE savings in training and inference costs. But what about the new trillion token “retrieval database” they added to the architcture? Surely that must add back some computational costs, balancing the cosmic seesaw?
Does infinity exist? This comment from MathOverflow riled my feathers a bit:
In the machine learning research community, achieving state-of-the-art usually means reporting a single score (percentage accuracy or F1) on a public research dataset. There are two legitimate reasons to report a “SOTA score” in a research paper, besides gaming the system.1 The
I’ve been using VSCode as my defacto Typescript IDE for the last few months. I’m a heavy Emacs user, however, and it was only a matter of time until I attempted to get a similar experience via Emacs configuration, thereby continuing my quest to use Emacs as the sole interface to my computer.
TL;DR - the fundamental assumption of probability theory is one of ignorance. This assumption is too easy to break in most contexts and leads to unfounded confidence in conclusions.
One of the most useful things I’ve learned during my PhD is how to use ducttape, a research workflow management system. Like many good software tools, the mindset behind ducttape is more powerful than the code itself. In this post, I’ll try to motivate the research workflow management mindset and then give you a sense of how ducttape solves the problems I present.