Back in the dark ages of typing code into editors we were aided by squigglies under broken code, click-to-definitions links, and so on. That was powered by language servers and type checkers. Several harnesses now expose an LSP as a tool, on the reasonable premise that better code intelligence makes for a better agent. Models, [ ]
It is a truth universally acknowledged that an airline in possession of an airplane must be in want of engines to make it go. Yet, somewhat surprisingly, they don’t really buy engines. Rolls-Royce were the notable innovator here in selling not an engine, but instead what they call power by the hour: airlines pay a [ ]
One good way to annoy a neuroscientist is to compare an LLM to the brain. It’s appealing though! There are similarities! In infancy we take a complex fusion of sensory inputs and learn to make predictions in latent space, while in pre-training a stack of Transformers learn to predict which number SolidGoldMagikarp will say next [ ]
We talked about this sort of thing a bit before, but now the official Multi-Teacher On-Policy distillation paper is out, and its a pleasant read: “MOPD for Capability Integration in LLM Post-Training”. The problem MOPD is solving is composing a bunch of different capabilities into the same model. Normally you do this with RL, with [ ]
The basic job of an eval is let you judge how good your model is on a task. If enough people use the same eval we can use it to benchmark the relative performance of multiple models on a level playing field. All good, no drama. But building good benchmarks is hard! ImageNet was a [ ]
Pre-training any kind of good LLM is very, very expensive. Thankfully, we have scaling laws. Lilian Weng of Thinky writes: Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss decreases predictably as we scale up model size N, dataset size D, and [ ]
Back in 2022 and 2023 there were two big branches of machine learning happening at Meta. The LLM work that led to Llama was a clean, smooth stack of repeated Transformer modules; the recommendation systems graphs were, by contrast, terrifying. Luckily, the industry has remedied that state of affairs by making LLMs a lot more [ ]
When we started building LLMs, we mostly focused on them knowing things. They had information encoded in their weights, and they could spit it out when given sufficient prompts. But an agent doesn t just need to know things; it needs to combine several kinds of knowledge. A lot of that is still in the weights: [ ]
The capabilities in a large language model emerge, mysteriously, from the training data. Everyone agrees that you start with a big pile of data, add some compute, and at the end you can vibe code. Opinions differ on what that pile of data should look like. Microsoft AI recently released an incredibly in-depth technical report [ ]
There has been a lot of drama about distillation: how (closed) frontier models are being used by other labs to boost their own performance on particularly hard tasks. The drama is not fake, exactly. Anthropic, and recently OpenAI, have a notable lead in the agentic-coding domain, and some of that is from having data that [ ]
A thing you can do is take the most performance and correctness sensitive part of your stack and just ask a chatbot to write it for you. They will sometimes get it right! Back towards the end of 2024 Ouyang et al at Stanford attempted to benchmark how often that happened with KernelBench. DeepSeek R1 [ ]
SIMT offered a fantastic bargain. You write a straight-line program, the machine runs a lot of copies of it, and when one waits for memory the hardware swaps in others. You look with disdain on the less enlightened thread programmers dealing with deadlocks and concurrency etc. etc. Choosing what to run where and when is [ ]
If you want to see what a very painful couple of months looks like for an ML research team, FAIR’s logbook of the OPT-175 pretraining from 2021 should top your list. The first few runs are basically: At each point the team tweaks some of the hyperparameters: learning rates, weight decay, clipping and so on, [ ]
Recently I had a conversation with an infrastructure team supporting an ML modeling group. The two orgs used to collaborate to ship experiments: the modeling team would come up with ideas, the infra team would augment their frameworks and build out tooling to make those ideas scalable. Together, they would ship an experiment every couple [ ]
You may have noticed that FlashAttention 4 was supported in PyTorch really quickly. That required a bit of new infrastructure: torch.native by Simon Layton. Prior versions of FlashAttention were written in Cutlass/C++, but for FA4 the team implemented the kernel in CuteDSL. Edit: Simon kindly pointed the FA4 work integration work predated his formalization of [ ]