Giles' blog
I recently overtrained a couple of GPT-2 style models , training them both on 40 tokens per parameter rather than the 20 per parameter that is generally regarded as "Chinchilla-optimal". The normal heuristic is that instead of doing that, you should scale up the number of tokens and the number of parameters equally -- so I would have been better off scaling up the model by 2 and the token count by…
Inspired by this LessWrong post , I thought I'd write about how I use AI here. This is less in the interest of disclosure, more to provide a snapshot of what I'm doing right now so that I can revisit it in the future and see how it changes. And hey, maybe it'll be of interest to you, dear readers. If I were to summarise my working philosophy in fewer than ten words, it would be: AIs identify…
The GPT-2-style models that I've been training work really well, and I've even managed to train some that perform better than the original OpenAI small model in terms of cross entropy loss on a test set. But as I wrote previously , there's a mystery: why do they perform worse on my instruction fine-tuning evaluation? I had various theories about why that might be, and to me, the most…
I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post . While I was writing up the results of my first experiment into possible causes, I ran the post past ChatGPT -- I always use an "editorial board" of AIs to check my posts for flow, style, and any technical errors (though all writing is always…
When I finished my project training an LLM from scratch , I was left with a minor mystery. Why were my models worse at instruction-following than the original OpenAI GPT-2 small weights? I had an evaluation that I was running, based on the instruction fine-tuning code in chapter 7 of " Build a Large Language Model (from Scratch) ". The process was to train a model on samples from the Alpaca…
I mentioned I'd got a new RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool. He's right that it's not really been my thing -- I've been focusing on my own LLMs…
When I was building my GPT-2 implementation in JAX , I started with just token embeddings for the input, and a separate output head (as I was not using weight tying ). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train…
For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, perry . I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX ), but there are a couple of problems. perry is my daily driver. If he's doing a training run, then everything is just a little…
This post is the capstone of the most long-running series on my blog . In December 2024 (!), I started reading Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ", and worked through it carefully. Being who I am, despite trying to apply a strict "no side quests" policy, I found myself zooming off and digging into all kinds of things. It's time to wrap it up. I had decided…
For over a year, I've been using Sebastian Raschka 's book " Build a Large Language Model (from Scratch) " -- and the multitude of side-projects that have branched out from reading it -- as something like a curriculum for learning about modern AI. The one final task I had set myself was to build and train an LLM from scratch just using my notes -- no reference to the book, no reference to the…