Continual Pre-training GPT‑OSS‑120B
1. GPT‑OSS‑120B Design
1.1 MoE Shape
GPT‑OSS‑120B is a decoder-only sparse Mixture-of-Experts Transformer according to the released GPT‑OSS‑120B config. The residual stream is 2,880 wide, the vocabulary contains 201,088 tokens, and 36 decoder blocks sit between an untied input embedding and LM head. Every block contains shared attention plus a router and 128 distinct feed-forward experts. Routing is token-wise and layer-wise: each token selects four experts in every block, so only 3.125% of a layer’s expert pool is evaluated for that token.
The underlying decoder-block vocabulary—masked self-attention, multi-head projections, position-wise FFNs, residual paths, and normalization—is developed step by step in the earlier Self-Attention Layer and The Transformers Architecture article. GPT‑OSS retains that Transformer skeleton while replacing each dense feed-forward sublayer with a routed expert pool and using its GPT‑OSS-specific attention, normalization, and positional choices.
| Component | Exact logical parameters | Derivation |
|---|---|---|
| One expert | 24,891,840 | gate/up: 2 × 2,880 × 2,880 + 5,760 bias; down: 2,880 × 2,880 + 2,880 bias |
| 128 experts in one layer | 3,186,155,520 | 128 × one expert |
| Attention in one layer | 26,550,144 | Q/K/V/O weights and biases plus 64 attention sinks |
| Router in one layer | 368,768 | [128, 2,880] weight + [128] bias |
| Two RMSNorms in one layer | 5,760 | 2 × 2,880 |
| All 36 decoder blocks | 115,670,886,912 | 36 × 3,213,080,192 |
| Embedding + untied LM head | 1,158,266,880 | 2 × 201,088 × 2,880 |
| Model total | 116,829,156,672 | Exact sum from the released weight index |
Experts contain 114.702B parameters—98.18% of the logical model—while all attention layers contain 0.956B, the two vocabulary matrices 1.158B, and routers plus norms about 0.013B. These totals come from the published tensor index and configuration. The commonly reported 5.1B active parameters per token can be reproduced more precisely as 36 × (four experts + attention + router + two norms) + one dense vocabulary projection + final norm = 5,132,849,472. The input embedding is a row lookup, so counting the whole embedding matrix again would not describe per-token compute.
1.2 Attention and Context
Attention uses grouped-query attention according to the released config: 64 query heads but only 8 key/value heads, each with head dimension 64. Q is projected from 2,880 to 4,096; K and V each project to 512; the output projection maps 4,096 back to the 2,880-wide residual stream. Sharing each KV head across eight query heads reduces KV-cache size. Human-numbered odd layers use 128-token sliding causal attention and even layers use full causal attention, producing 18 local and 18 global layers. The local layers control long-context cost, while every other full layer still permits global information flow.
Position information uses RoPE with YaRN scaling in the GPT‑OSS‑120B config: original context 4,096, scale factor 32, maximum position 131,072, rope_theta=150000, beta_fast=32, and beta_slow=1. Each layer also has 64 learned attention-sink scalars—one per query head—which give softmax probability a learned non-content destination instead of forcing all mass onto ordinary tokens. Attention projections include biases, attention dropout is zero, and RMSNorm uses epsilon 1e‑5.
For the geometric derivation of RoPE and the PI/NTK/YaRN context-extension family, see RoPE and M‑RoPE.
The router maps each normalized token state from 2,880 dimensions to 128 logits, chooses the top four, and softmax-normalizes those four logits to weight the sum of their outputs, matching num_local_experts=128 and num_experts_per_tok=4. The fused expert projection interleaves 2,880 gate and 2,880 up values. Its exact gated activation clamps gate to a maximum of 7, clamps up to ±7, and computes (up + 1) × gate × sigmoid(1.702 × gate); a 2,880 × 2,880 down projection then returns to the residual width. The released configuration sets swiglu_limit=7.0 and router_aux_loss_coef=0.9.
1.3 MXFP4 Weights
OpenAI’s GPT‑OSS repository states that the models were post-trained with MXFP4 quantization of the MoE weights, making GPT‑OSS‑120B run on a single 80GB-class accelerator; Hugging Face’s MXFP4 docs describe it as the GPT‑OSS-specific 4-bit floating-point path in Transformers. In this article, MXFP4 should be read as the released model’s inference/storage weight format for MoE experts, not as a full optimizer-state format for CPT or full fine-tuning.
MXFP4 is a microscaling format: the Microscaling Data Formats paper defines MX as a per-block scale combined with narrow per-element formats; MXFP4 uses 4-bit FP4 elements and an 8-bit shared scale. In the common OCP-style layout, a block stores 32 E2M1 FP4 values plus one E8M0 scale, giving 32 × 4 + 8 = 136 bits per block, or about 4.25 bits/value before metadata and packing overhead. The block scale preserves more dynamic range than plain INT4 because each local group can choose its own power-of-two scale.
MXFP4 does not make full training fit in 80GB. The released checkpoint’s quantization and sharding index converts expert projection blocks to MXFP4 but excludes attention, routers, embeddings, and the LM head. This explains inference on an approximately 80GB accelerator; full training still needs dequantized or master weights, gradients, optimizer states, activations, communication buffers, and checkpoints.
1.4 Design Choices
Why can head_dim=64 work? A useful community hypothesis applies Scientific Spaces’ n > 8.33 ln N dimensional heuristic, motivated by minimum entropy and the connection between attention and skip-gram. With the natural logarithm used by that article, the estimates are about 40.4 for a 128-token window, 69.3 for 4,096 tokens, and 98.2 for 131,072 tokens. Thus 64 is a plausible conventional dimension for a local 128-token head, but it does not satisfy the same heuristic for GPT‑OSS’s full-attention layers or 128k context. The original community calculation used log2 in places, which is not the convention evidenced by the source’s numerical examples. More importantly, this is a heuristic—not proof that OpenAI selected 64 for this reason. The released architecture compensates systemically: 64 query heads give Q width 4,096 despite a 2,880 residual width, eight KV heads keep the cache narrow, and local/full attention alternates. The related MLA analysis is helpful for understanding why larger head dimensions may add representational room, but “larger is always better” ignores head count, cache size, kernels, and the fixed compute budget.
hidden_size = intermediate_size = 2,880 is not a dense MLP ratio of four. The official MoE forward pass sends the same normalized token to four independently parameterized experts, applies each expert’s gated nonlinearity and down projection, then takes a router-weighted sum. It does not concatenate the four experts into one 11,520-wide dense FFN. It is still useful to say that one token evaluates four separate 2,880-wide intermediate branches—11,520 intermediate scalar activations in aggregate—and therefore pays roughly the parameter/compute of four same-width experts, but the routing and nonlinear mixture are mathematically different from one width-11,520 layer. The common ReLU rank-loss argument in the linked FFN expansion discussion assumes a particular independent-zero model and does not transfer directly to GPT‑OSS’s clipped gated-SwiGLU. Superposition and the Transformer-circuits framework motivate richer feature spaces, but do not prescribe an exact ratio. In practice, dimensions should also be benchmarked on the target stack: hardware/model co-design reports up to 39% throughput differences between shape choices at comparable parameter counts, so an A100 microbenchmark result such as an MLP ratio near 2.7 is hardware-, kernel-, dtype-, and batch-specific rather than universal.
Biases and clipping are real; the optimizer story is unknown. The released reference code contains router, fused gate/up, and down-projection biases. It also clamps the gate at 7, clamps the up branch to ±7, adds 1 to the up branch, and evaluates (up + 1) × gate × sigmoid(1.702 × gate). Bounding expert activations is plausibly friendly to low-precision expert weights and outliers, but OpenAI has not published that causal rationale. Likewise, the presence of MLP biases does not establish whether Muon or any other optimizer was or was not used; the original optimizer and state remain undisclosed.
The receptive field is global every other layer. The official mask construction applies a 128-token sliding window to zero-indexed even layers and full causal attention to odd layers. Multiplying 128 × 36 = 4,608 is therefore not GPT‑OSS’s receptive field. Even in a hypothetical all-local stack, a careful token-count bound is approximately 1 + L × (W − 1), not simply L × W; in the real alternating stack, each full layer can already attend to the entire allowed prefix, up to the configured 131,072 positions. Local layers reduce attention cost between global layers rather than limiting the whole model to roughly 4k context.
Attention sinks are not KV shifting or four sink tokens. In OpenAI’s reference SDPA, the model owns 64 learned sink scalars—one per query head—concatenates one sink logit to each attention-score row, performs softmax, then discards the sink probability before multiplying by V. There is no learned sink K/V vector and no sink_token=4; four is the number of active MoE experts. KV Shifting Attention instead shifts key/value states to make induction easier and is a different mechanism. The sink gives attention a learned “none of the content tokens” destination, while KV shifting changes the content-bearing K/V computation.
UE8M0 is the scale behind MXFP4. The OCP MX specification defines an MXFP4 block as 32 E2M1 FP4 values sharing one E8M0 scale: 136 bits, or 4.25 bits per weight. NVIDIA calls the scale ue8m0: an unsigned exponent-only 8-bit float with no mantissa or infinity, with 0xff reserved for NaN and packed as ue8m0x2 in PTX. Newer Blackwell block-scaled MMA paths can consume MXFP4 with scale factors directly; other runtimes may dequantize or use different kernels. Finally, “FP4 is only deployment quantization” is incomplete for GPT‑OSS: the model card says OpenAI post-trained with MoE weights quantized to MXFP4. That does not mean original pre-training or optimizer states were FP4, but it is more than an after-the-fact deployment conversion.
1.5 Qwen3.5 Contrast
Qwen3.5‑35B‑A3B is a useful sparse-MoE contrast because it targets similar active compute with a very different backbone. It is an Apache‑2.0, natively multimodal causal model with a vision encoder, 35B total parameters, approximately 3B activated parameters, a 2,048-wide text stream, 248,320 padded vocabulary entries, and 40 text layers. Its published layout is 10 × [3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)]: 30 recurrent linear-attention layers alternate with ten full-attention layers instead of GPT‑OSS’s 18 sliding-window and 18 full-attention layers. The released model is post-trained; the separate Base checkpoint is the pre-trained starting point intended for further training.
| Property | Qwen3.5‑35B‑A3B | GPT‑OSS‑120B |
|---|---|---|
| Modalities | Native text, image, and video | Text only |
| Logical / active parameters | 35B / ≈3B | 116.8B / ≈5.13B |
| Depth and width | 40 layers, width 2,048 | 36 layers, width 2,880 |
| Sequence mixer | 30 Gated DeltaNet + 10 gated full-attention layers | 18 sliding-window + 18 full-attention layers |
| MoE per layer | 256 routed experts; top 8 + one shared; expert width 512 | 128 routed experts; top 4; expert width 2,880 |
| Native context | 262,144; YaRN extension to 1,010,000 | 131,072 via YaRN from 4,096 |
| Conversation contract | Qwen template, <think>, multimodal/tool tags | Harmony roles, channels, recipients, and effort |
The exact Qwen3.5 config makes the hybrid block concrete. Gated DeltaNet uses 32 value heads and 16 Q/K heads at dimension 128 plus a four-wide local convolution. Full gated attention uses 16 query heads, two KV heads, head dimension 256, output gating, and a 64-dimensional rotary subspace with interleaved multimodal RoPE sections 11/11/10 and rope_theta=10,000,000. Every layer has 256 routed experts, chooses eight, and also evaluates one shared expert; routing therefore combines sparse specialization with a dense path available to every token, with router_aux_loss_coef=0.001. Input and output embeddings are untied. One MTP layer predicts additional future tokens during training. The vision tower has 27 layers, width 1,152, intermediate width 4,304, 16 heads, 16×16 spatial patches, temporal patch size two, and 2× spatial merging before projection to the 2,048-wide language stream.
The release config stores BF16 tensors, whereas the Qwen3.5 training report describes a native FP8 training pipeline that applies low precision to activations, MoE routing, and matrix multiplications while retaining BF16 in sensitive operations. That is training arithmetic, not the same artifact as GPT‑OSS’s MXFP4 expert-weight representation. The same report attributes Qwen3.5’s throughput to early-fusion text/image/video training, heterogeneous parallelism for vision and language, a 248k multilingual vocabulary, and sparse computation; it reports support for 201 languages and dialects. Those are Qwen3.5-family facts, not a fully reproducible 35B-specific optimizer recipe.
The official model card reports both language and vision evaluations: for example, Qwen3.5‑35B‑A3B scores 85.3 on MMLU‑Pro, 84.2 on GPQA Diamond, 69.2 on SWE‑bench Verified, 81.4 on MMMU, and 71.1 on AndroidWorld under Qwen’s stated setups. These are vendor-reported numbers, not an independent apples-to-apples ranking; their value here is to show that the 3B-active model targets reasoning, coding, agents, and vision jointly rather than text perplexity alone.
2. Training Regimes
2.1 Scope and Goals
OpenAI released GPT‑OSS as open-weight models with native Harmony usage, and the GPT‑OSS‑120B weights/config are public; OpenAI has not released the original optimizer state, terminal learning rate, total pre-training tokens, or an official continued-pretraining recipe. The values below are engineering starting points triangulated from direct GPT‑OSS fine-tuning recipes and public large-model training runs. GPT‑OSS‑120B has about 117B total parameters from the released tensor shapes and roughly 5.1B active parameters per token; its MXFP4 80GB inference footprint does not imply that full-parameter AdamW training fits on one 80GB GPU.
“Mid-training” often conflates four optimization regimes. Continued pretraining changes knowledge, language, or domain distribution through next-token prediction on raw text and code. SFT teaches instruction following, format, tool use, and task mappings. DPO learns preference boundaries from chosen/rejected pairs. Online RL learns a policy from rollouts and rewards, as in the open Agent Factory 3 GPT‑OSS experiments. Model size alone cannot determine the learning rate.
| Stage | Training signal | Primary goal | Do not copy |
|---|---|---|---|
| CPT / domain adaptation | Raw text, code, documents | Change knowledge or domain distribution | SFT/DPO sequence batches |
| Full SFT | Supervised conversations | Behavior, format, tools | Million-token CPT batches |
| LoRA SFT | Supervised conversations | Low-cost behavior adaptation | Full-parameter LR assumptions |
| DPO | Chosen/rejected pairs | Preference alignment | CPT-scale learning rates |
| Online RL / RLVR | Rollouts and rewards | Agent policy | Static-dataset batch descriptions |
2.2 LR and Scale
Evidence anchors: Axolotl GPT‑OSS‑120B full SFT, OpenAI GPT‑OSS‑20B LoRA SFT, Tülu 3 SFT/DPO recipes, Reuse, Don’t Retrain CPT, Code Llama CPT, and OLMo 3 mid-training.
| Purpose | Initial / peak LR | Schedule | Global batch | Source basis |
|---|---|---|---|---|
| Narrow CPT, <5B tokens | 5e‑6–2e‑5; start 1e‑5 | Cosine; 1–2% warmup for a reset optimizer; finish near peak/30–peak/100 | 0.25–1M tokens/update | Reuse + Code Llama extrapolation |
| Domain CPT, 20–100B tokens | 1e‑5–4e‑5; start 2e‑5 | Cosine; 0.5–1% warmup | 1–4M tokens/update | OLMo 100B, Code Llama, Reuse |
| Full SFT | 2e‑6–2e‑5; broad SFT starts at 5e‑6–1e‑5 | Linear/cosine or constant+warmup; ~3% | 16–128 sequences/update | Axolotl 120B + Tülu 70B |
| LoRA SFT | 5e‑5–2e‑4; start 1e‑4 | Cosine; 3% warmup; optionally retain 10% final LR | 8–32 sequences/update | OpenAI GPT‑OSS LoRA |
| DPO | 1e‑7–5e‑7; start 2e‑7 | Linear; 10% warmup | 64–128 pairs/update | Tülu 3 DPO |
| 120B agentic RL | Undisclosed for public full runs; LoRA pilot 1e‑6–5e‑6 sweep | Algorithm-specific | Report rollout/policy tokens | Agent Factory 3 |
For domain CPT, run a short 1e‑5 → 2e‑5 → 4e‑5 sweep: this is an engineering interpolation between Axolotl’s direct GPT‑OSS‑120B SFT anchor, Reuse, Don’t Retrain’s scheduler study, and large CPT examples such as Code Llama. A true continuation with optimizer/scheduler state may need no warmup, but released GPT‑OSS weights do not provide that state; a short warmup is safer when AdamW is reinitialized. Track domain validation, general retention, instruction following, and downstream performance—not training loss alone.
Record unique examples or pairs, consumed tokens, and disk bytes separately. This mirrors the separation used by public data/recipe releases such as Tülu 3, OLMo 3, and Code Llama. The following storage column is only tokens × 4 bytes for int32 token IDs, not raw JSON/Parquet size.
| Training tokens | int32 token IDs | Typical interpretation | External anchor |
|---|---|---|---|
| 1M | 0.004 GB | Small behavioral SFT | OpenAI GPT‑OSS LoRA |
| 100M | 0.4 GB | Medium post-training | Tülu 3 |
| 1B | 4 GB | Large SFT / small CPT pilot | Tülu 3 |
| 20B | 80 GB | Meaningful narrow-domain CPT | Reuse CPT |
| 100B | 400 GB | Formal mid-training ingredient | OLMo 3 |
| 500B–1T | 2–4 TB | Large capability shift or specialization | Code Llama |
Behavioral SFT may need only 1k–50k high-quality examples, with OpenAI’s GPT‑OSS cookbook showing a 1k-example LoRA target; broad instruction tuning often uses 100k–1M conversations, with Tülu‑3 70B SFT as the full-parameter post-training anchor. A 0.1–5B-token CPT pilot tests feasibility, while 20–100B tokens is a credible main range for 70B/120B domain adaptation, anchored by OLMo 3’s 100B-token mid-training mix and the Reuse, Don’t Retrain CPT study. At 4M tokens/update, 100B tokens gives about 25,000 optimizer updates; using that batch on a 1B-token pilot gives only 250 updates.
2.3 Batch and Retention
global sequences = DP ranks × microbatch × gradient accumulation, matching the effective-batch definition in the Tülu 3 reproduction guide. With tensor or expert parallelism, DP ranks are not physical GPU count. For CPT, report the sum of non-padding tokens per optimizer update because packing and padding make sequence counts ambiguous; large CPT anchors such as Code Llama and OLMo 3 should be compared in tokens/update, not examples/update.
On 8×A100‑80GB, validate LoRA/SFT, Harmony labels, and loss correctness first; Axolotl’s published 8-card full-parameter run uses H100s plus CPU offload. On 32×A100‑80GB, FSDP2/ZeRO‑3 full FFT and CPT are more realistic. Extra GPUs should primarily reduce accumulation and wall-clock time rather than trigger linear LR scaling.
Do not feed domain data exclusively until the end. Use general replay, capability-related replay, staged mixtures, or checkpoint merging, following the retention concerns in Reuse, Don’t Retrain and the mixed-data design in Code Llama. Select mixture switches using both the LR trajectory and retention evaluations. A domain gain accompanied by immediate general-loss degradation is a signal to lower LR or increase replay—not simply to train on more tokens.
2.4 Qwen3.5 Training
The public evidence supports a training outline, not an exact recipe. According to the official Qwen3.5 report and model card, pre-training uses early fusion: text, image, and video are converted into a common token sequence and optimized together, with expanded multilingual, STEM, coding, and reasoning data under stricter filtering. The hybrid Gated DeltaNet/full-attention language backbone, vision encoder, sparse MoE, routers, and MTP objective are trained as one foundation model. Qwen reports a decoupled heterogeneous-parallel system that overlaps the vision and language components, near-text-only throughput for mixed multimodal training, and native FP8 execution with BF16 retained where numerical sensitivity requires it.
Post-training then adds instruction behavior, explicit thinking, tool use, visual reasoning, and agent policy. Qwen’s release description says reinforcement learning was scaled across million-agent environments with progressively harder task distributions; its asynchronous framework disaggregates rollout inference from training and adds dynamic load balancing and fault recovery across text, multimodal, and multi-turn environments. This tells us the direction—large heterogeneous SFT/RL and environment scaling—but not the exact 35B‑A3B SFT mixture, teacher models, reward composition, RL algorithm, rollout count, accepted-token count, optimizer, learning rate, batch, or total pre-training tokens. Those values are not publicly specified; the older Qwen3 four-stage recipe and larger Qwen3.5 variants must not be presented as the 35B checkpoint’s recipe.
For new CPT, start from Qwen3.5‑35B‑A3B‑Base when the goal is domain knowledge, then recover chat/tool behavior with the exact Qwen template. Starting from the post-trained checkpoint is defensible when preserving its existing instruction policy matters more than a clean base-model objective, but it creates the same forgetting/alignment tradeoff discussed for GPT‑OSS. A text-only CPT run may freeze the 27-layer vision tower; a joint multimodal CPT run must preserve image/video packing, multimodal RoPE positions, modality balance, and vision-language projection updates. These are engineering choices, not undisclosed Qwen settings.
3. Harmony Format
3.1 Roles and Channels
GPT‑OSS was post-trained on Harmony for conversation structure, reasoning, and function calls; the official Harmony guide states that self-hosted inference or training stacks must preserve this format. Harmony mirrors the conceptual shape of the Responses API and applies the hierarchy system > developer > user > assistant > tool.
| Role | Official purpose | Source |
|---|---|---|
system | Model identity, knowledge cutoff/current date, reasoning effort, valid channels, and built-in tools | Harmony guide |
developer | What other formats call the system prompt: instructions, function tools, and response formats | Harmony guide |
user | User input | Harmony guide |
assistant | Reasoning, tool calls, preambles, and final output | Harmony guide |
tool | A tool result; the concrete tool name becomes the message author/role | Harmony guide |
| Channel | Meaning | Operational rule | Source |
|---|---|---|---|
analysis | Raw CoT and internal tool use | Do not expose it to users; it is not trained to the same safety standard as final output | Harmony guide |
commentary | Function calls and user-visible preambles | Usually where functions are called; preambles may be displayed | Harmony guide |
final | End-user response | Display this channel to users | Harmony guide |
The system message should keep the standard model identity, include knowledge cutoff and current date, set Reasoning: low|medium|high (medium is the default), and declare analysis, commentary, final as valid channels, as specified in the OpenAI Harmony article. If developer-defined functions exist, it should say that calls to the functions namespace go to commentary. The developer message begins with instructions and may add # Tools or # Response Formats. Product identity/persona changes belong in developer instructions, not by rewriting the system identity.
Prefer the official openai_harmony renderer from PyPI or crates.io. It supplies typed system/developer content, converts JSON Schema tools to Harmony declarations, renders conversations to tokens, parses completion tokens back into messages, and provides a streaming parser exposing the current role, channel, content type, recipient, delta, and accumulated content. Streaming parsing also avoids breaking Unicode while decoding token-by-token.
encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
prompt_tokens = encoding.render_conversation_for_completion(conversation, Role.ASSISTANT)
# Run the model; exclude the stop token from new_tokens.
messages = encoding.parse_messages_from_completion_tokens(new_tokens, Role.ASSISTANT)
stream = StreamableParser(encoding, role=Role.ASSISTANT)| Special token | ID | Meaning |
|---|---|---|
<|start|> | 200006 | Start a message/header |
<|end|> | 200007 | End a fully stored message |
<|message|> | 200008 | Header-to-content boundary |
<|channel|> | 200005 | Channel field in the header |
<|constrain|> | 200003 | Tool-input content type, such as JSON |
<|return|> | 200002 | Stop inference after a completed response |
<|call|> | 200012 | Stop inference to execute a tool |
3.2 Calls and State
A complete stored message is <|start|>{header}<|message|>{content}<|end|>, following the Harmony wire format. A completion begins at an assistant header, may emit multiple analysis messages, and ends with either <|return|> or <|call|>. The former is a decode-time stop token: normalize it to <|end|> before persisting the reply into the next prompt. Keep <|return|> at the end of supervised targets, while prior messages in a prompt should end in <|end|>.
Function tools live in the developer message under # Tools, normally inside a functions namespace, as shown in the OpenAI Harmony function-call examples. The official convention is TypeScript-like: a no-argument function is type name = () => any; argument-bearing functions receive a single _ object; comments describe fields; return type remains any; and definitions are separated by blank lines. Staying close to this representation improves tool-call accuracy.
<|channel|>analysis<|message|>Need current weather.<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_current_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
<|start|>functions.get_current_weather to=assistant<|channel|>commentary
<|message|>{"sunny":true,"temperature":20}<|end|>
<|start|>assistantThe recipient may appear in the role or channel portion of the header according to Harmony parsing rules. After <|call|>, execute the function, append a tool-authored commentary message addressed to assistant, and resume sampling. A commentary message without a recipient can be a user-visible preamble—for example, an action plan before several calls—so it must not be treated as hidden CoT.
Conversation-state handling has two cases. After an assistant turn reaches final, remove its old analysis before the next user turn and keep the normalized final message, consistent with Harmony’s state guidance. During an unfinished tool/function chain, however, preserve the preceding analysis, call, and tool result when resuming generation; the model needs that chain to continue reasoning correctly. This distinction must be represented explicitly in SFT and agent trajectories.
3.3 Outputs and Loss
A structured response format belongs at the end of the developer message under # Response Formats, with a format name, optional description, and JSON Schema, as specified in the Harmony response-format section. Prompting the schema influences behavior but does not guarantee conformance; strict adherence requires grammar/constrained decoding during sampling.
GPT‑OSS was released with built-in tool behavior, and the Harmony guide describes built-in browser and Python tools in the system message rather than developer tools. Browser calls normally use the analysis channel with recipients such as browser.search, browser.open, and browser.find; Python calls use analysis with recipient python. Function tools defined by the developer normally use commentary. A runtime must still tolerate occasional channel variation described by the official guide.
For SFT, mask system, developer, user, and tool-output tokens; train current assistant analysis, commentary/calls, preambles when desired, and final output. This loss policy is an engineering choice derived from the Harmony channel semantics, OpenAI’s GPT‑OSS SFT notebook, and common supervised post-training practice in Tülu 3. Preserve all three reasoning efforts across comparable task families. For domain knowledge injection, begin with final loss weight 1.0 and analysis weight 0.2–0.5, or train analysis only on verifier-filtered traces. Validate raw messages → Harmony rendering → token IDs → labels → parsed messages, including <|return|>/<|end|> normalization and tool-resume exceptions.
Large-scale CPT should remain raw text/code causal-LM training, matching continued-pretraining work such as Reuse, Don’t Retrain, Code Llama, and OLMo 3; do not convert tens of billions of tokens into fake chats. Follow CPT with a smaller Harmony recovery SFT covering ordinary QA, low/medium/high reasoning, no-tool and tool trajectories, preambles, structured outputs, multi-turn final history, tool-chain continuation, and instruction conflicts.
3.4 Qwen3.5 Format
Qwen3.5 must not be rendered with Harmony. Its published chat template is ChatML-like: messages are delimited by <|im_start|> and <|im_end|>, the system message must come first, and ordinary roles are system, user, and assistant. There is no Harmony developer role, no analysis/commentary/final channel header, no recipient field, and no native Reasoning: low|medium|high control. Instead, enable_thinking=true opens <think>; disabling it emits an empty thinking block before the answer. The template can consume a separate reasoning_content field or split it from assistant content.
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Explain the routing decision.<|im_end|>
<|im_start|>assistant
<think>...current-turn reasoning...</think>
...final answer...<|im_end|>The template deliberately removes old reasoning before a new user query while keeping prior final answers, closely paralleling Harmony’s “do not replay completed CoT” rule even though the wire formats differ. For SFT, mask system/user prompts and tool responses; train the current assistant’s thinking plus final answer or tool call only when the trace is trusted. Do not train synthetic long thinking merely to increase length. Pair task difficulty with an appropriate reasoning budget, validate the split around <think>...</think>, and test multi-turn rendering so previous hidden traces cannot leak back into the prompt.
Tools are serialized differently as well. JSON tool schemas are injected into the system message under # Tools and <tools>; assistant calls use <tool_call><function=...><parameter=...>, and results appear inside <tool_response>. Multimodal content uses <|vision_start|><|image_pad|><|vision_end|> or the corresponding video pad. The Base model card notes that control tokens were already included during pre-training so parameter-efficient tuning need not modify the very large embedding matrix. Keep two separate render/parse test suites: one for Harmony/GPT‑OSS and one for Qwen3.5.
4. Training Practice
4.1 Runbook
For the general mechanics behind this parameter policy—how hard top‑k routing is trained, which experts receive gradients, and why “all parameters are trainable” does not mean that every weight changes on every step—see Which Parameters Update? in the companion MoE article. For GPT‑OSS‑120B full CPT this means the shared blocks, router, and all 128 experts per layer are trainable, while each token supplies a task gradient only to its selected four experts.
The parameter distribution from the published tensor index determines which adaptation method changes what. Attention-only LoRA touches a small shared subnetwork and is a strong first baseline for policy, tool use, and agent behavior, as shown by Agent Factory 3. Expert/MLP adaptation has much greater capacity for domain knowledge and generation-distribution shifts, but it also introduces routing imbalance and far larger optimizer/checkpoint costs, which is why the Axolotl 120B recipe emphasizes offload and checkpoint storage. Training the router changes which frozen or trainable experts receive tokens; freezing it stabilizes paths but may limit adaptation to a new domain. Always log per-layer expert selection frequency, routing entropy, tokens per expert, dropped/overflowed tokens if the implementation has capacity limits, and auxiliary router loss alongside domain and retention evaluations.
| Public anchor | Configuration | Training lesson |
|---|---|---|
| Axolotl GPT‑OSS‑120B full SFT | LR 2e‑5, BF16, FSDP2, activation checkpointing, CPU offload | Sharding/offload and checkpoint storage dominate full 117B optimization |
| OpenAI GPT‑OSS‑20B LoRA SFT | LR 2e‑4, 3% warmup, global batch 16 | Validate Harmony, loss, and selected target modules before scaling |
| Agent Factory 3 GPT‑OSS‑120B RL | Attention LoRA, frozen MoE/router, asynchronous rollout and routing replay | Policy improvement can be tested without updating the 114.7B expert subsystem |
- Lock model/tokenizer/config revisions; verify BF16/dequantized logits against the released MXFP4 checkpoint and test attention-sink plus sliding/full-attention parity.
- Establish domain, general, Harmony, tool, and routing-utilization baselines; run a LoRA or 0.5–1B-token CPT pilot before full optimization.
- Sweep CPT LR 1e‑5/2e‑5/4e‑5 and replay ratio; select the domain-gain/retention/routing-stability frontier, then scale toward 20–100B tokens using Reuse, Code Llama, and OLMo as scale anchors.
- Run high-quality Harmony recovery SFT, followed only then by preference tuning or RL.
For full training, use BF16, activation checkpointing, FSDP2/ZeRO‑3 or expert parallelism, sharded state dictionaries, and kernel parity tests, aligned with the Axolotl GPT‑OSS guide and large-model practice in Tülu 3. The internal LLM memory accounting guide derives why weights, gradients, optimizer states, and activations—not checkpoint weight size alone—determine feasibility, and its ZeRO/FSDP section explains the sharding vocabulary used here. Save the exact parameter-target policy, router loss, expert statistics, model/tokenizer/Harmony revisions, dataset hashes, packing and token counts, DP/TP/EP topology, microbatch, accumulation, optimizer/scheduler/RNG state, software commit, quantization/dequantization path, and attention-kernel version. Retain best-domain, best-retention, and best-routing-stability checkpoints rather than only the final step.
4.2 Qwen3.5 Runbook
Qwen3.5’s 256 routed experts are all trainable in a full run, but one token’s task gradient reaches only its selected eight routed experts plus the shared expert; routers, attention/DeltaNet blocks, embeddings, vision-language projection, and MTP head are shared or separately exercised. This is the same sparse-update distinction derived in Which Parameters Update?. Log per-layer expert load, routing entropy, shared-versus-routed contribution, auxiliary routing loss, dropped tokens if capacity is finite, and utilization by modality and language. A globally balanced router can still collapse locally on image tokens or a new domain.
- Choose and pin the Base or post-trained revision, processor, config, and template. Decide explicitly whether the vision tower and MTP head train.
- Before scale-up, test BF16/FP8 logit and gradient parity; Gated DeltaNet recurrent state and convolution; full-attention mRoPE; image/video position IDs; MoE dispatch; and MTP loss. The released config’s BF16 dtype is not proof that every training kernel used BF16.
- Run a short text-only or multimodal CPT sweep and select on domain gain, general retention, multimodal retention, routing stability, and long-context evaluation. Do not copy GPT‑OSS LR merely because both are sparse MoEs; architecture, checkpoint stage, active paths, and optimizer continuity differ.
- Restore behavior with Qwen-native SFT, then validate thinking-on/off, tools, image/video turns, multilingual generation, and 262k native context before DPO or asynchronous agent RL. Use YaRN extension toward 1,010,000 only as a separately evaluated deployment mode described by the official model card.
The main reproducibility boundary is simple: the official report reveals architecture-level training choices—early fusion, hybrid parallelism, native FP8, MTP, and asynchronous RL scale—while the 35B‑A3B release does not reveal a complete data/optimizer schedule. A credible experiment therefore records its own unique samples and consumed tokens, text/image/video mixture, sequence packing, trainable modules, MTP coefficient, router coefficient, FP8 recipe, DP/TP/EP topology, rollout versions, and every evaluation revision rather than labeling an inferred configuration “the Qwen recipe.”
For artifact validation, the official serving guide supports Transformers, vLLM, SGLang, and KTransformers, uses the qwen3 reasoning parser and qwen3_coder tool parser, and exposes MTP-assisted speculative decoding. Text-only deployment can omit the vision model, but a text-only training export should still be checked against the original processor, vocabulary, untied LM head, and chat template before it is considered compatible.