Glossary of Fine-Tuning Methods Explained

Note: AI Weekender has moved. New posts are published at ai-weekender.com, and this Substack is now an archive.

To keep receiving weekly issues, please subscribe at ai-weekender.com instead of here.

Last week I covered when fine-tuning is worth it.

This week is a deep dive into fine-tuning methods and the vocabulary that comes with them, from SFT and LoRA to RLHF and GRPO.

This guide maps each method to what it optimizes and when you would run it.

Here’s a quick lookup you can skim through:

Read on in the following sections for more detailed explanations.

Fine-tuning methods differ in:

• Objective they optimize

• Data format that objective requires

All of them shift what the model tends to output for a given input:

• Supervised fine-tuning (SFT) trains on examples where each prompt has one correct answer, so you need (input, correct output) pairs. LoRA and QLoRA are popular ways to run SFT on limited hardware.

• Preference-based training optimizes relative quality. For the same prompt, one response is marked as preferred over another. The dataset looks like (prompt, chosen, rejected). RLHF and DPO both implement this.

• Reinforcement learning (RL) with a verifiable reward doesn’t require human labels and uses a reward function you define. It is a rule that returns a quantitative score for each completion. That function dictates what RL maximizes, which could be an exact label match, or a math proof, or any objective you can score automatically. An example of RL is GRPO.

For most AI products, SFT is the default way to customize outputs, from taxonomy to tone.

Sections below go from what most product teams try first (i.e. SFT / QLoRA) toward heavier alignment (DPO, RLHF, GRPO) and full fine-tuning.

SFT teaches the model to produce your answer when it sees a similar input. You train on labeled (input, correct output) pairs and it learns to match those outputs on new prompts.

Instruction tuning is SFT where each example is a short chat, consisting usually of a user message and an assistant reply.

PEFT (parameter-efficient fine-tuning) freezes the pre-trained base and trains a small adapter on top. At deploy time you ship base weights + adapter (for example LoRA or QLoRA).

LoRA is an efficient fine-tuning method that keeps each layer’s weight matrix W frozen and learns a small add-on instead of rewriting W directly.

It adds two small trainable matrices, B and A, to change how the layer behaves instead of updating the full W matrix. At inference, the effective weights are W′ = W + BA.

Only A and B receive gradient updates. On a 7B model, that means training tens of millions of parameters vs. billions for full fine-tuning.

QLoRA is for when the full model will not fit in GPU memory.

QLoRA is still LoRA on top, but compresses the frozen base from the standard 16-bit (bf16) weight storage to 4-bit (NF4) storage to use roughly 4x less VRAM. The adapters stay in bf16 and receive gradients.

For a 7B model, base weight memory drops from ~14GB (stored in bf16) to ~4GB (stored in 4-bit NF4), so it can be fine-tuned on a 16GB laptop or a single T4 instead of a 24GB+ setup.

Preference training applies when you are choosing a relatively better answer, not necessarily a single objectively correct one.

Each training row is a triple (prompt, chosen, rejected): two completions for the same prompt, with one rated higher. The model learns to favor chosen over rejected.

RLHF aligns a model to human preferences when there is no single correct answer per prompt. The pipeline trains a reward model on preference labels, then fine-tunes with RL.

This process is expensive because it requires:

• Labels at scale

• Heavy RL compute

• Multiple training stages

DPO is the lighter alternative when you already have preference triples.

It uses one offline training loop to nudge the LLM toward chosen over rejected outputs, without needing a reward model.

GRPO (Group Relative Policy Optimization) fine-tunes an LLM with a defined scoring rule. This is a function that returns a numeric score for each completion, with no human preference labels or learned reward model.

For each prompt, the model samples a batch of answers, scores them, and learns to favor ones that beat the batch average. GRPO works best when you use scoring functions the RL loop can maximize, such as correct math answers, passing tests, or exact label match.

It is a strong fit for code and math reasoning models. DeepSeek-R1 was trained this way.

FFT updates every parameter in the base model, not just an adapter. It gives maximum flexibility to reshape representations, but costs far more VRAM and is a longer training process.

FFT is for domain-specific foundation models and teams that already run custom training clusters. It requires in-house expertise and infrastructure to train, version, and serve a full checkpoint.

When you are deciding what to run, start with the labels you have (or can realistically collect):

• Input–output pairs / one correct answer per prompt: run SFT with QLoRA, or LoRA if you have the VRAM. That is the default for most product work to have LLMs output specific taxonomy, and stable voice or register.

• Pairwise preferences: Each row is (prompt, chosen, rejected), where chosen is preferred, not necessarily the only correct answer. Common methods of preference training include DPO (offline training loop) and RLHF (multi-stage pipeline with a reward model that is more common at foundation-lab scale).

• Score completions with an objective rule (i.e. math verification, tests pass, exact match): GRPO-style RL can be effective. That is mostly foundation-lab work on code and math reasoning models such as DeepSeek-R1.

Which fine-tuning method have you tried and what was the impact of the fine-tuned model?

Note: AI Weekender has moved. New posts are published at ai-weekender.com, and this Substack is now an archive.

To keep receiving weekly issues, please subscribe at ai-weekender.com instead of here.