Understanding Go AI Inference: What is Inference?

Welcome to a new series! For most developers today, using a large language model means one thing: an HTTP call to somebody else’s computer. You send a prompt to an API, tokens come back, and everything in between is somebody else’s magic.

But here’s what I find much more interesting: you can run these models locally, inside your own process, on your own hardware — and if you’re a Go developer, you can do it directly from Go. Two projects make this genuinely pleasant today: Yzma, which lets Go call the llama.cpp libraries directly (without cgo — that gets its own article later in the series), and Kronk, which builds a high-level, OpenAI-API-feeling SDK and model server on top of Yzma. And underneath both of them sits llama.cpp, the C/C++ inference engine that made running LLMs on ordinary hardware practical.

In this series we’re going to understand that whole stack from the inside: what llama.cpp actually does when it “runs a model,” how Yzma manages to call it from Go, and how Kronk turns all of that into an engine you’d actually want in production.

📌 A note on scope

The goal of this article is to understand the mechanics around the model — how a model file is stored, loaded, and run — not what happens inside the neural network itself. The math inside those layers is a whole topic of its own, and it’s out of scope here.

The llama.cpp details are against master as of July 2026 (commit ad8d8219) — llama.cpp moves fast and doesn’t do stable releases, so I’m pinning a commit instead of a version. And this is deliberately a conceptual tour: we’re not going to get into the C++ code details. We’ll also stay out of the Go layers entirely — Yzma and Kronk get their own articles.

With the scope set, let’s dive in.

What Inference Actually Is

Let’s start by demystifying the word. A language model has two very different lives.

The first life is training: you show the model terabytes of text, and an optimization process gradually adjusts billions of numbers — the weights (also called parameters) — until the model gets good at one narrow task: given a sequence of text, predict what comes next. Training takes months and data centers. We’re not talking about training in this series, at all.

The second life is inference: training is over, the weights are frozen, and now you just… use them. The best mental model is a plain, pure function: it takes two inputs — your sequence of text and those billions of frozen weights — and returns one output, a prediction for the next token. Something like nextToken(input, weights). That’s it. That’s the whole secret. No learning happens, nothing is updated — the weights are read-only arguments, so a model file is exactly as smart the millionth time you call it as the first.

📌 A note on weights

If you’re curious about how much memory those weights actually take up — how quantization shrinks them, and why they usually dominate the VRAM you need to run a model locally — Kronk’s VRAM calculator write-up is a nice read.

So when you download a “model,” what you’re downloading is essentially a giant pile of frozen numbers — those weights — plus a description of how to use them. And “running” the model means doing a very large amount of arithmetic between your input and those numbers. If that sounds suspiciously hand-wavy, don’t worry: we’ll demystify the weights later in the article, when we crack open an actual model file. For now, all you need to hold on to is that a model is a huge pile of numbers that somebody else already calculated for you.

We’ve already mentioned tokens a couple of times now — but what exactly is a token? Let’s see.

Tokens: the model’s alphabet

There’s one catch before any arithmetic can happen: models don’t read text. They work with tokens — integer IDs from a fixed vocabulary, where each ID corresponds to a chunk of text (a word, a piece of a word, a punctuation mark).

Our prompt “The capital of France is " might become something like five or six token IDs — say [791, 6864, 315, 9822, 374] (the exact IDs depend entirely on the model’s vocabulary). From this point on, the model never sees letters again; everything downstream is done in terms of these integers.

So the model receives a list of integers. What does it actually do with them?

One pass, one prediction

Here’s the shape of the computation, stripped to its essence. One important thing up front: the model doesn’t process the input word by word in separate rounds — the whole sequence of tokens goes through the network together, in a single pass, and out comes a single prediction. Let’s walk through it with this diagram:

Reading it left to right, we start by converting each token ID into an embedding — a vector of numbers that represents the token in a form the model can work with (if you’d like to understand embeddings better, Weaviate has a nice primer). Then all of those vectors are fed into the model together, which runs a ton of calculations and ends up producing a list of scores — one per token in the vocabulary — saying how likely each token is to come next. And from that list, we pick one: " Paris”.

That final pick has a name — sampling — and we’ll come back to how it really works later. But notice something crucial first: all this work produced just one token. How do we get a whole sentence out of a machine that only predicts the next word?

The autoregressive loop

By doing it again. And again. This is the autoregressive part, and it’s what makes an LLM conversation tick. Let’s see how it works with this diagram:

Follow the arrows. We feed our sequence — “The capital of France is " — into the model, it does one pass, and we pick a token: " Paris”. Here’s the trick that turns a single prediction into a whole answer: instead of stopping, we append " Paris" to the sequence and run the model again, now on “The capital of France is Paris”. The next token might be “.”, we append that too, and around we go — one new token per lap — until the model picks a special end-of-generation token (its way of saying “I’m done”), or we hit a length limit and stop it ourselves.

Every word of every LLM answer you’ve ever read was produced one token at a time by this loop. Generation isn’t one big computation — it’s the same next-token prediction, in a for loop.

So much for the theory. Before we can watch that loop actually run, everything hinges on one thing we’ve been hand-waving about: those frozen weights. Where do they come from, and what does a “model” actually look like on disk? Time to open the box.

GGUF: The Whole Model in One File

Time to demystify those weights. That pile of frozen numbers isn’t a shapeless heap: it’s organized into named arrays called tensors (a tensor is just an n-dimensional array — a matrix generalized to more dimensions) — for example, a single layer of the neural network is typically made up of several tensors. GGUF (GGML Universal File) is how that bag of tensors is stored on disk, and its whole design follows from one goal: a single, self-describing file. One .gguf holds everything needed to run the model — weights, architecture, hyperparameters, the entire tokenizer, even the chat template — with no sidecar files to download. Copy one file, run the model. (The very largest models sometimes get split into a few .gguf shards for practicality, but the principle holds.)

So what does one of these files actually look like inside? The format is documented right at the top of the GGUF header in ggml — the tensor library llama.cpp is built on, and where the file format itself lives (ggml/include/gguf.h). But rather than paste the raw spec, let me draw it for you:

Let’s read it top to bottom. Right at the start sits a tiny header — the magic bytes GGUF (so a program can check “yes, this really is a GGUF file”), a version number, and two counts telling the reader how many tensors and how many metadata entries to expect. Everything after that falls into three sections, and the rest of this section is really just a guided tour of those three boxes: metadata, tensor descriptors, and the data blob. Let’s take them one at a time.

The metadata: a model that describes itself

That’s box 1 in the diagram. The metadata is a flat list of typed key-value pairs — the values are simple scalars, strings, and homogeneous arrays. The keys are namespaced strings, and there are a few namespaces (the ones you can spot in the diagram) I want to call your attention to:

• general.* — identity: general.architecture says which kind of model this is (llama, qwen2, gemma, …). This one key decides which graph-building code llama.cpp will use to set up the inference machinery.
• <arch>.* — hyperparameters, prefixed by the architecture name: llama.embedding_length (the size of those token vectors), llama.block_count (how many layers), llama.context_length (how many tokens fit in the window), etcetera…
• tokenizer.* — the entire tokenizer (the piece responsible for turning our text into tokens) lives inside the model file. tokenizer.ggml.tokens is the full vocabulary as a string array, alongside the tokenizer’s configuration parameters and even tokenizer.chat_template — the template that turns a chat conversation into a flat prompt string.

When llama.cpp opens a file, it reads general.architecture first, and everything else follows from there.

That’s the what of the model. Now let’s see where the actual numbers live.

The blob: getting the weights into memory

Now for boxes 2 and 3 in the diagram, which work as a pair. Box 2 is the tensor descriptors: one row per tensor with its name, shape, and type — but no data. Instead, each row carries an offset, and that’s the red arrow in the picture: the offset points at where that tensor’s bytes actually live inside box 3, the data blob.

The interesting part is how those bytes get into memory. The loader (llama_model_loader, src/llama-model-loader.cpp) first parses only the header, metadata, and tensor descriptors — no tensor data is read at all. That’s enough to know what goes where: some layers stay on the CPU, others go to the GPU, and each destination handles its bytes its own way. For CPU tensors, llama.cpp by default doesn’t read() gigabytes of weights into freshly allocated buffers — it maps the file into the process’s address space with mmap and points each tensor directly at mapping_address + tensor_offset. For GPU tensors that trick doesn’t work — VRAM is a different memory altogether — so their bytes really do get copied: uploaded from the file into GPU buffers, once, at load time.

So now we have the file’s tensors sitting in memory, each in its right place. Time to run something.

The KV Cache: Why Generation Stays Fast

Let’s start with what we already have: our model’s data, the part that never changes. Loading it hands us the llama_model — the file itself, in memory: the weights, the tokenizer, all the read-only stuff. Because nothing about it changes, it can be shared — one loaded model, many conversations, and those multi-gigabyte weights sit in memory only once.

Now we need the dynamic data, and that’s what the llama_context is for: it represents a single conversation, and it’s where all the changing state lives — the working space for the calculations and, above all, the KV cache. Compared to loading a model, creating a context is cheap, so you spin up one per conversation.

So what is this KV cache? At every step of the loop we hand the model an ever-longer sequence, and reprocessing it from scratch on every pass would be hugely wasteful. Avoiding that is the job of the KV cache.

The trick is that while processing a token, the model computes some intermediate values for it that never change afterwards. So instead of recomputing them, llama.cpp stores them in the KV cache and reuses them. Each new token only has to do its own work and read the rest from memory — so every generated token costs roughly the same, no matter how long the text has already grown.

This is also why a bigger context window eats more memory: llama.cpp reserves cache space for the whole window up front, when the context is created, and fills it in token by token as the conversation grows (some architectures use sliding-window attention to soften this). And it’s why there’s a short pause before the first word — the model is filling the cache for your whole prompt.

With that trick in mind, we’ve followed a prompt all the way through the model: tokens in, a list of scores out, repeated one token at a time for as long as we keep generating. But a list of scores still isn’t a word — something has to turn it into one.

Sampling

The model leaves us with the logits — one score per vocabulary token. Something still has to pick one, and that’s the job of the sampler. It’s a separate, pluggable stage that runs after all the heavy math is done, on that single row of scores.

The simplest possible sampler just takes the highest score — that’s greedy sampling, and for our prompt it’s perfect: " Paris" wins every time. But always taking the top choice makes open-ended writing dull and repetitive, so real setups run the scores through a sampler chain first: a pipeline of small steps, each one reshaping or trimming the list of candidates before handing it to the next. The common ones, left to right in the diagram:

• Penalties push down tokens that appeared recently, to stop the model repeating itself.
• Top-k keeps only the k highest-scoring candidates and throws the rest away.
• Top-p keeps the smallest group of top candidates whose probabilities add up to p, dropping the long tail of unlikely tokens.
• Temperature flattens or sharpens the scores. Turn it down and the top choice dominates even more (safe, predictable); turn it up and the field evens out (more adventurous, more random).
• Grammar forbids any token that would break a required format. This is the trick behind “give me valid JSON”: tokens that would produce invalid output are simply made unpickable right here.

Whatever the chain, it ends the same way: one token comes out — " Paris". And what do we do with it? Append it and run the model again. The loop is closed.

At this point we have every piece of the puzzle on the table. Let’s watch them work together.

Walking Through an Example

Let’s put every piece in order by running “The capital of France is " end to end, the way llama.cpp’s own minimal example (examples/simple/simple.cpp) does it.

Step 0 — Load. We open some-model.gguf. The loader reads the magic, version, metadata, and tensor descriptors — a few kilobytes. From general.architecture and friends it builds the llama_model; from tokenizer.* it builds the vocabulary. The weights each go to their place: CPU tensors by default end up pointing straight into the mmap’d file, and GPU tensors are uploaded to VRAM. Then we create a llama_context, which sets aside the memory inference will need — the KV cache, plus some scratch space for the calculations.

Step 1 — Tokenize. llama_tokenize() turns "The capital of France is " into token IDs using the vocabulary that came inside the GGUF file — say, five tokens at positions 0–4.

Step 2 — Prompt processing. We hand the five tokens to llama_decode(), which runs them through the model. The keys and values for all five tokens land in the KV cache, and we get back one row of scores: one logit per token in the vocabulary, for “what comes after The capital of France is.”

Step 3 — Sample. The sampler chain runs over that row: penalties adjust the scores, top-k and top-p trim the candidate list, temperature reshapes it, and out of what’s left one token gets picked: Paris. The chain accepts it (penalties and grammars take note), and we detokenize it back to text.

Step 4 — Loop. From here we just repeat: append " Paris” and run llama_decode again — this time computing only the new token, since everything before it is already in the KV cache — sample the next token, and keep going until the model emits its end-of-generation marker. One cheap pass per word, and the text streams out. (llama.cpp’s own simple.cpp is this exact loop in about thirty lines, if you want to see it in code.)

When you eventually chat with a model through Kronk’s OpenAI-style API, this exact loop is what’s spinning at the bottom of the stack.

And that’s the machinery, top to bottom. Time to zoom out.

Summary

So let’s step back and put the whole picture together.

At heart, inference is just running your input through a huge pile of frozen numbers — no learning, no magic, the same weights every time. And it works one token at a time: the model looks at everything so far and predicts the next token, we tack it on, and we run it again. That loop is the whole show.

A single run isn’t as mysterious as it sounds either. Your text gets chopped into tokens, each token becomes a list of numbers the model can crunch, the whole sequence flows through together, and out the other end comes a score for every possible next token. Something then has to pick one of those scores — that’s sampling, and it’s really a little assembly line of knobs (penalties, top-k, top-p, temperature, grammars) that shape the choice before a token pops out.

All of that lives in one tidy GGUF file — weights, settings, and the entire tokenizer in a single self-describing blob. When it loads, every weight goes to its place: CPU tensors by default end up pointing straight into the mmap’d file, and GPU tensors get uploaded to VRAM. From there you work with just two things: the read-only model you can share across conversations, and a per-conversation context that holds everything that changes. And the reason a long conversation doesn’t grind to a halt is the KV cache — the model remembers the work it already did for earlier tokens, so each new word costs about the same no matter how far along you are.

That’s the engine, end to end.

Now that we know what the engine underneath actually does, we can climb the stack. Next up is Yzma, where we’ll see how it leverages llama.cpp to run inference from Go. That’s where this series turns properly Go-flavored. See you there!