Building an AI Text Detector From Scratch
An End-to-End Project With Dataset Construction, Model Training, Local Deployment, and RLVR
Sebastian Raschka, PhD
I'm an LLM Research Engineer with over a decade of experience in artificial intelligence. My work bridges academia and industry, with roles including senior staff at an AI company and a statistics professor. My expertise lies in LLM research and the development of high-performance AI systems, with a deep focus on practical, code-driven implementations.
Latest posts
How Claude's Text Watermarking Works
Short illustration of how Claude's text watermarking is supposed to work based on Anthropic's released materials.
Build a Reasoning Model From Scratch Is Now on Amazon
Short note on the Amazon availability of Build a Reasoning Model From Scratch and a warning about counterfeit black-and-white copies sold through Amazon India.
Muse Glimmer 30B Architecture Notes
Short architecture note on Meta Muse Glimmer 30B, including gated local and global GQA, KV-cache efficiency, and release-time benchmark comparisons.
LLMs From Scratch Reaches 100,000 GitHub Stars
Short note celebrating the LLMs-from-scratch repository passing 100,000 GitHub stars and summarizing its learning materials.
Kimi K3 Architecture Notes
Short architecture note on Kimi K3, including LatentMoE, Kimi Delta Attention, Attention Residuals, NoPE, multimodality, and inference-efficiency choices.
A Few Notable Open-Weight Models This Week
Short note on the architectures of six new open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3.
Correction for Listing 6.5 in Build a Reasoning Model From Scratch
Short correction note for the random seed in Listing 6.5 on page 198 of Build a Reasoning Model From Scratch.
Controlling Reasoning Effort in LLMs
How LLMs Learn Low-, Medium-, and High-Effort Reasoning Modes
Inkling: A New Open-Weight 975B MoE with a Few Surprises
Architecture and benchmark notes on Thinking Machines Lab's 975B Inkling MoE, including short convolutions, relative-position bias, training, and effort control.
200,000 Subscribers
Short note celebrating Ahead of AI reaching 200,000 subscribers.
GPT 5.6 Has 72 Possible Configurations. What's A Good Default?
Short note on how GPT 5.6 model and effort choices map onto training-time and inference-time scaling, producing 72 configurations.
Build a Reasoning Model From Scratch Is Out
Short note announcing the release of Build a Reasoning Model From Scratch and linking the publisher and Amazon pages.
Using Local Coding Agents
Short note linking a new article on setting up local coding agents with open-weight models.
Using Local Coding Agents
Using Open-Weight Models in Local Coding Harnesses as an Alternative to Claude Code and Codex Subscriptions
Local Open-Weight LLMs in Coding Harnesses
Short note on trying local open-weight LLMs across Qwen-Code, Codex, and Claude Code harnesses.
GLM-5.2 and IndexShare for Long-Context Sparse Attention
How GLM-5.2 uses IndexShare to reuse sparse-attention token selections across layers and reduce long-context indexer computation.
VibeThinker-3B and the Strength of Post-Training
A closer look at VibeThinker-3B, a Qwen2.5-Coder-based model whose reported reasoning gains come from a detailed post-training pipeline.
North Mini Code and Agentic Coding Benchmarks
Architecture and benchmark notes for North Mini Code, Cohere's 30B-A3B MoE trained for repository, terminal, and code-generation tasks.
LLM Research Papers: The 2026 List (January to May)
A curated roundup of notable LLM research papers that came out this year
Nemotron 3 Ultra and Latent MoE Scaling
Architecture notes on Nemotron 3 Ultra, including its 108-layer hybrid stack, Latent MoE scaling, NVFP4 recipe, MTP, and inference results.
MiniMax M2 Technical Report Notes
Technical notes on MiniMax M2, including full attention, fine-grained MoE routing, agent training data, speed rewards, and self-evolution.
DeepSeek Sparse Attention From Scratch
Short note on a DeepSeek Sparse Attention from-scratch implementation added to the LLMs-from-scratch repository.
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs
Implementing LLM Architectures From Scratch
Short note linking a talk on implementing LLM architectures from scratch and comparing new open-weight model implementations against references.
My Workflow for Understanding LLM Architectures
A learning-oriented workflow for understanding new open-weight model releases
Components of A Coding Agent
How coding agents use tools, memory, and repo context to make LLMs work better in practice
Gemma 4 Architecture and Benchmark Notes
Architecture and benchmark notes for Gemma 4 31B and 26B-A4B, including hybrid attention, long-context changes, and evaluation caveats.
LLM Architecture Gallery Diff Tool
Compare two LLM architectures side by side across attention, decoder type, layer recipe, model scale, context length, and KV-cache use.
A Visual Guide to Attention Variants in Modern LLMs
From MHA and GQA to MLA, sparse attention, and hybrid architectures
Inside the LLM Architecture Gallery
Browse 93 LLM architecture diagrams with model fact sheets, source links, concept explainers, and a side-by-side comparison tool.
Nemotron 3 Super Throughput Notes
Nemotron 3 Super combines Mamba-2, Latent MoE, sparse GQA, and shared-weight MTP in a throughput-oriented 120B-A12B model.
A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026
A Round Up And Comparison of 10 Open-Weight LLM Releases in Spring 2026
State of AI 2026 with Sebastian Raschka, Nathan Lambert, and Lex Fridman
I recently sat down with Lex Fridman and Nathan Lambert for a comprehensive 4.5 h interview to discuss the current state of progress of AI, and what the...
Categories of Inference-Time Scaling for Improved LLM Reasoning
Inference scaling has become one of the most effective ways to improve answer quality and accuracy in deployed LLMs. The idea is straightforward. If we are...
The State Of LLMs 2025: Progress, Problems, and Predictions
A 2025 review of large language models, from DeepSeek R1 and RLVR to inference-time scaling, benchmarks, architectures, and predictions for 2026.
LLM Research Papers: The 2025 List (July to December)
A curated list of LLM research papers from July–December 2025, organized by reasoning models, inference-time scaling, architectures, training efficiency...
From Random Forests to RLVR: A Short History of ML/AI Hello Worlds
Two years ago, I posted a list of Hello World examples for machine learning and AI on social. Here, the Hello World means beginner-friendly examples to...
From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates
Similar to DeepSeek V3, the team released their new flagship model over a major US holiday weekend. Given DeepSeek V3.2's really good performance (on GPT-5...
Recommendations for Getting the Most Out of a Technical Book
This short article compiles a few notes I previously shared when readers ask how to get the most out of my building large language model from scratch books...
Beyond Standard LLMs
After I shared my Big LLM Architecture Comparison a few months ago, which focused on the main transformer-based LLMs, I received a lot of questions with...
DGX Spark and Mac Mini for Local PyTorch Development
The DGX Spark for local LLM inferencing and fine-tuning was a pretty popular discussion topic recently. I got to play with one myself, primarily working...
Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)
Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples
Understanding and Implementing Qwen3 From Scratch
Previously, I compared the most notable open-weight architectures of 2025 in The Big LLM Architecture Comparison. Then, I zoomed in and discussed the...
From GPT-2 to gpt-oss: Analyzing the Architectural Advances
OpenAI just released their new open-weight LLMs this week: gpt-oss-120b and gpt-oss-20b, their first open-weight models since GPT-2 in 2019. And yes, thanks...
The Big LLM Architecture Comparison
It has been seven years since the original GPT architecture was developed. At first glance, looking back at GPT-2 (2019) and forward to DeepSeek-V3 and...
LLM Research Papers: The 2025 List (January to June)
The latest in LLM research with a hand-curated, topic-organized list of over 200 research papers from 2025.
Understanding and Coding the KV Cache in LLMs from Scratch
KV caches are one of the most critical techniques for efficient inference in LLMs in production. KV caches are an important component for compute-efficient...
Coding LLMs from the Ground Up: A Complete Course
Why build an LLM from scratch? It's probably the best and most efficient way to learn how LLMs really work. Plus, many readers have told me they had a lot...
The State of Reinforcement Learning for LLM Reasoning
A lot has happened this month, especially with the releases of new flagship models like GPT-4.5 and Llama 4. But you might have noticed that reactions to...