# model scratch (blogs) — RSS Amplifier Recent posts from the 2 feeds in the RSS Amplifier directory that cover model scratch. Page: Feed: --- ## [Use the built-in GELU, don't roll your own!](https://www.gilesthomas.com/2026/08/built-in-gelu) _2026-08-20 · Giles' blog_ Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training the… ## [My experience at a Deftones concert in Vienna](https://karlosmid.com/2026/08/my-experience-at-deftones-concert-in-vienna/) _2026-08-17 · Karlo Šmid Blog_ A short travel and concert report from seeing Deftones live in Vienna on August 16, 2026. ## [A quick(ish) Chinchilla check](https://www.gilesthomas.com/2026/08/chinchilla-check) _2026-08-07 · Giles' blog_ I recently overtrained a couple of GPT-2 style models , training them both on 40 tokens per parameter rather than the 20 per parameter that is generally regarded as "Chinchilla-optimal". The normal heuristic is that instead of doing that, you should scale up the number of tokens and the number of parameters equally -- so I would have been better off scaling up the model by 2 and the token count by… ## [How I use AI on this blog](https://www.gilesthomas.com/2026/07/ai-use) _2026-07-31 · Giles' blog_ Inspired by this LessWrong post , I thought I'd write about how I use AI here. This is less in the interest of disclosure, more to provide a snapshot of what I'm doing right now so that I can revisit it in the future and see how it changes. And hey, maybe it'll be of interest to you, dear readers. If I were to summarise my working philosophy in fewer than ten words, it would be: AIs identify… ## [Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining](https://www.gilesthomas.com/2026/07/why-do-openai-gpt2-weights-beat-mine-3-overtraining) _2026-07-31 · Giles' blog_ The GPT-2-style models that I've been training work really well, and I've even managed to train some that perform better than the original OpenAI small model in terms of cross entropy loss on a test set. But as I wrote previously , there's a mystery: why do they perform worse on my instruction fine-tuning evaluation? I had various theories about why that might be, and to me, the most… ## [Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix](https://www.gilesthomas.com/2026/07/why-do-openai-gpt2-weights-beat-mine-2-the-bugfix) _2026-07-30 · Giles' blog_ I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post . While I was writing up the results of my first experiment into possible causes, I ran the post past ChatGPT -- I always use an "editorial board" of AIs to check my posts for flow, style, and any technical errors (though all writing is always… ## [Why do OpenAI's GPT-2 weights beat mine?](https://www.gilesthomas.com/2026/07/why-do-openai-gpt2-weights-beat-mine-1-intro) _2026-07-29 · Giles' blog_ When I finished my project training an LLM from scratch , I was left with a minor mystery. Why were my models worse at instruction-following than the original OpenAI GPT-2 small weights? I had an evaluation that I was running, based on the instruction fine-tuning code in chapter 7 of " Build a Large Language Model (from Scratch) ". The process was to train a model on samples from the Alpaca… ## [Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090](https://www.gilesthomas.com/2026/07/benchmarking-qwen-3-6-35b-moe-rtx-3090) _2026-07-24 · Giles' blog_ I mentioned I'd got a new RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool. He's right that it's not really been my thing -- I've been focusing on my own LLMs… ## [Building intuition about LLM parameter counts](https://www.gilesthomas.com/2026/07/llm-parameter-counts) _2026-07-10 · Giles' blog_ When I was building my GPT-2 implementation in JAX , I started with just token embeddings for the input, and a separate output head (as I was not using weight tying ). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train… ## [poppy the training box, part 1: the beginnings](https://www.gilesthomas.com/2026/07/poppy-the-training-box-1-the-beginnings) _2026-07-09 · Giles' blog_ For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, perry . I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX ), but there are a couple of problems. perry is my daily driver. If he's doing a training run, then everything is just a little… ## [AI QA that opens fix PRs (Sponsored)](https://crawlproof.com/a/CNe2DIYcTeii) _2026-07-09 · **Sponsored**_ Run a browser QA from plain-English instructions; get GitHub issues and optional fix PRs. ## [Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)](https://www.gilesthomas.com/2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax) _2026-07-08 · Giles' blog_ This post is the capstone of the most long-running series on my blog . In December 2024 (!), I started reading Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ", and worked through it carefully. Being who I am, despite trying to apply a strict "no side quests" policy, I found myself zooming off and digging into all kinds of things. It's time to wrap it up. I had decided… ## [Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run](https://www.gilesthomas.com/2026/06/llm-from-scratch-34a-building-a-jax-training-loop-for-an-llm-training-run) _2026-06-30 · Giles' blog_ For over a year, I've been using Sebastian Raschka 's book " Build a Large Language Model (from Scratch) " -- and the multitude of side-projects that have branched out from reading it -- as something like a curriculum for learning about modern AI. The one final task I had set myself was to build and train an LLM from scratch just using my notes -- no reference to the book, no reference to the… ## [Recap on fine-tuning to follow instructions](https://karlosmid.com/2026/06/recap-on-fine-tuning-to-follow-instructions/) _2026-06-17 · Karlo Šmid Blog_ Workbook notes on instruction fine-tuning a pretrained GPT-2 model in Elixir. ## [Fine-tuning to follow instructions](https://karlosmid.com/2026/06/fine-tuning-to-follow-instructions/) _2026-06-15 · Karlo Šmid Blog_ Fine-tuning a pretrained GPT-2 model in Elixir so it can follow instructions, generate task-specific responses, and be evaluated with Ollama. ## [Recap of fine-tuning for classification](https://karlosmid.com/2026/06/recap-on-fine-tuning-for-classification/) _2026-06-12 · Karlo Šmid Blog_ Workbook notes on fine-tuning a pretrained GPT-2 model for SMS spam classification in Elixir. ## [Fine-tuning for classification](https://karlosmid.com/2026/06/fine-tuning-for-classification/) _2026-06-05 · Karlo Šmid Blog_ This post fine-tunes a pretrained GPT-2 model for SMS spam classification in Elixir. ## [Recap of training a GPT model](https://karlosmid.com/2026/06/recap-on-training-GPT-model/) _2026-06-02 · Karlo Šmid Blog_ Workbook notes on GPT training, cross-entropy loss, perplexity, and model weights. ## [Pretraining on unlabeled data in Elixir](https://karlosmid.com/2026/05/pretraining-on-unlabeled-data-in-elixir/) _2026-05-29 · Karlo Šmid Blog_ A practical Elixir walkthrough of GPT pretraining: calculating generation loss, training on The Verdict, controlling decoding with temperature and top-k sampling, saving checkpoints, and loading OpenAI GPT-2 weights. ## [Estimating the population of Zabok using public data](https://karlosmid.com/2026/05/estimating-the-number-of-population-for-zabok-using-public-data/) _2026-05-14 · Karlo Šmid Blog_ The last public population census in Croatia was done in 2021. With the latest development in Zabok, I made a simple estimate of the current population of Zabok. ## [Recap on implementing GPT model from scratch](https://karlosmid.com/2026/05/recap-on-implementing-GPT-model-from-scratch/) _2026-05-08 · Karlo Šmid Blog_ TL;DR I am recaping blog post Implementing a GPT model from scratch by answering workbook questions Build llm from scratch chapter 4, and reading Giles blog posts on this chapter: part15 part16 part17 part18 part19 Giles’ blog posts I got lost during Giles’ part 15 on the first read. The second read made things clearer. He goes deep into the GPTModel architecture and how data flows from tokens to… ## [Ship software peacefully (Sponsored)](https://crawlproof.com/a/zEI5dTCjTVYN) _2026-05-08 · **Sponsored**_ Connect your repo and deploy instantly — Railway handles config, scaling, and monitoring. ## [Implementing a GPT model from scratch](https://karlosmid.com/2026/05/implementing-a-GPT-model-from-scratch/) _2026-05-02 · Karlo Šmid Blog_ This post implements a GPT-style model from scratch in Elixir/Nx. ## [AWS error: not authorized to perform](https://karlosmid.com/2026/05/amazon-error-not-authorized-to-perform/) _2026-05-01 · Karlo Šmid Blog_ A misleading Amazon Connect authorization error turned out to be an API request formatting issue. Comparing the call with AWS CLI output and checking CloudTrail revealed that StartTime and EndTime needed epoch timestamp values.