17.1. dasLLAMA LLM inference: models, sessions, generation, chat

CPU large-language-model inference in pure daslang: load a GGUF model, tokenize, run the transformer, sample — or hold a full chat — validated token-for-token against llama.cpp on every supported family. Run with -jit; examples/dasLLAMA/run.das and chat.das show the canonical program shape.

Supported model families (GGUF — fp32 / f16 / q8_0 / q4_0 / mxfp4 weights read directly; K-quant files such as Q4_K_M / Q5_K_M / Q6_K run on native K-quant kernels):

  • Llama — Llama-2 / TinyLlama, Llama-3.1 / 3.2, Mistral-7B-Instruct, SmolLM2, plus llama2.c .bin checkpoints

  • Qwen — Qwen2.5, Qwen3 (QK-norm), Qwen3.5 / Qwen3.6 (hybrid Gated-DeltaNet attention, incl. the 35B-A3B MoE); MoE: Qwen1.5-MoE (routed + sigmoid-gated shared expert), Qwen3-30B-A3B (routed-only, renormalized top-k); vision + omni: Qwen3-Omni-30B and dense Qwen3-VL (M-RoPE image positions, deepstack), Qwen2.5-Omni / VL (window-attention ViT) — images and audio in chat

  • Phi — Phi-3.5-mini

  • Gemma — Gemma-2, Gemma-3 (per-layer sliding-window patterns, vision via the SigLIP mmproj), Gemma-4 (12B / 31B dense, the 26B-A4B MoE, and the E2B / E4B edge series with per-layer embeddings + cross-layer KV sharing; vision and audio in chat via the family mmprojs)

  • gpt-oss — gpt-oss-20b (attention sinks, native MXFP4 experts, YaRN long context, Harmony chat format)

The architecture is picked from GGUF metadata at load — the same program runs any of these.

Hands-on tutorials (overview): the problem statement, hello, generation, chat and templates, sampling, sessions and memory, performance, the architecture registry.

17.1.1. Type aliases

DlimImageInfo = DlimImageInfo

typedef DlimImageInfo = dasllama_image::DlimImageInfo aka DlimImageInfo

GpuTierStatus = GpuTierStatus

typedef GpuTierStatus = dasllama_gpu_tier::GpuTierStatus aka GpuTierStatus

GpuTierWant = GpuTierWant

typedef GpuTierWant = dasllama_gpu_tier::GpuTierWant aka GpuTierWant

GpuModelMarks = GpuModelMarks

typedef GpuModelMarks = dasllama_gpu_resident::GpuModelMarks aka GpuModelMarks

17.1.2. Types

The engine types the API below works with. They are created and consumed by the functions of this module; their remaining fields are engine implementation detail.

Model

A loaded model: weights, config, tokenizer, and the architecture’s blocks and chat template, as produced by load_model. User code touches config (e.g. cap config.seq_len before create_session on large-context models) and arch (the GGUF architecture name).

Session

One generation stream over a model: the KV cache, scratch buffers, sampling RNG, and the current position n_past. logits holds the distribution produced by the last eval. A model serves many independent sessions.

BatchWorkspace

Caller-owned scratch for eval_batch: the batched activation buffers a step of B sessions shares. One per concurrent batch, reused across calls; holds no session state — positions, caches and logits stay in the sessions.

KVPool

A caller-owned paged KV-cache pool (create_kv_pool): sessions created over it allocate fixed-size page groups on demand, so cache memory tracks the actual context instead of the full seq_len slab. One pool serves many sessions; an eval_batch batch must share one pool. Keep it alive and in place while its sessions live.

PrefixCache

A page-granular prefix cache over one pool’s sessions (create_prefix_cache): finished streams donate their KV pages keyed by a chained page hash of the token history, and later requests attach the longest cached prefix instead of re-prefilling it. Pages are refcounted with the pool; an LRU budget bounds retention.

KVDtype

Per-session KV-cache codec picked at create_session/create_chat: f16 — the default, half the KV bytes and faster deep-context decode (stores clamp to ±65504); f32 — the bit-exact reference; q8_0 — block-quantized (llama.cpp -ctk/-ctv q8_0), half the f16 bytes again and near-lossless in practice; needs head_size and kv_dim to be multiples of 32 (checked at create); tq4 — the FWHT-rotated ternary codec, smaller still.

QuantMode

Weight representation picked at load: fp32 — the token-exact reference; q8 — int8 quantization, the fast CPU path (K-quant/mxfp4/Q4_0 files keep their native planes on the same rails); q4_0 — the legacy requant tier, 4-bit blocks, smallest footprint.

SamplingParams

Sampling knobs: temp (<= 0 selects greedy argmax), top_k (0 = no cutoff), and repetition penalty (1.0 = none) applied over the last penalty_last_n generated tokens. The defaults are greedy — SamplingParams() reproduces argmax exactly.

Stats

Timing of the last generate/respond call: n_prompt/n_gen token counts, ttft_s (seconds to first token), and prefill_tps/gen_tps throughput in tokens per second.

LlmCaps

What the model honestly supports at the chat layer, as returned by caps: system_prompt is false for architectures with no system role (gemma), where the chat layer silently folds the system prompt into the first user turn. Grows as gaps surface.

ChatSession

A conversation over a model: its session, the resolved chat template, and the running transcript in history. Create with create_chat, then drive with add_user + respond.

ThinkStream

The incremental reasoning/content splitter for a streamed reply (make_think_stream): feed decoded pieces through think_feed, flush with think_finish. Holds partial reasoning markers back across chunk boundaries; a non-thinking family yields a pass-through stream.

ThinkSplit

One reply split at its reasoning boundary, as returned by split_reasoning: reasoning holds the thinking span (empty when the model answered directly), content the answer.

ToolReply

A fully classified tool-capable reply, as returned by parse_calls: the reasoning span, the content, and the calls the model made (empty when it answered directly).

ToolCall

One parsed tool call: the function name, its args normalized to JSON object text, and the wire id where the family carries one (mistral; empty elsewhere).

AudioTower

A loaded audio encoder: Whisper-family encoder weights plus the model-specific projector tail, as produced by load_audio_tower from an mmproj GGUF. Pass it to create_chat to enable add_user_audio turns.

VisionImage

A decoded RGB8 image: width, height, and the pixel bytes in rgb. Produced by load_image_rgb (the optional dasllama_vision_io companion) or built directly; add_user_image / encode_image letterbox and normalize it to the model’s geometry.

VisionEmbedder

A loaded vision embedder of whatever family the mmproj GGUF turned out to be (gemma4uv — the gemma-4 dense embedder; gemma4v — the gemma-4 E-series ViT tower; gemma3v — the gemma-3 SigLIP tower; qwen3v — the Qwen3-Omni / dense Qwen3-VL ViT, deepstack included; qwen25v — the Qwen2.5-Omni/VL window-attention ViT), as produced by load_vision_embedder, which sniffs the family. Pass it to create_chat to enable add_user_image turns; vision_proj_dim must match the decoder’s embedding width (checked at create).

VisionState

Caller-owned scratch for the embedder forward of whichever family is carried: the buffers encode_image reuses across calls. One per embedder user; holds no image state between calls.

AudioEmbedder

A loaded audio encoder of whatever family the mmproj GGUF turned out to be (gemma4a — the gemma-4 E-series Conformer), as produced by load_audio_embedder, which probes the file; audio_probe_proj_dim answers 0 where absence is an answer. audio_proj_dim must match the decoder’s embedding width. The server’s media worker owns one per armed slot.

AudioState

Caller-owned scratch for the audio encoder forward of whichever family is carried: the buffers encode_audio reuses across calls. One per embedder user; holds no clip state between calls.

DlimImageInfo

One prepared image beside a GGUF, as listed by dlim_inventory: file, path, bytes, the image version, its bake identity, and a verdict — CURRENT loads; STALE vN is an older image version; OTHER a different bake configuration or box; FOREIGN a different flavor.

GpuTierStatus

Snapshot of the GPU tier in force (gpu_tier_status): the device name, vram_bytes held, and the resident/streamed layer counts, dense planes and classifier residency the armed backend reports.

GpuTierWant

A programmatic GPU tier request (set_gpu_tier_want): auto_tier asks for the measured-best rail set; moe_layers/moe_stream/vram_mb and the per-rail flags override it. The core DASLLAMA_GPU_* env knobs override the matching fields when present. GpuTierWant() is a zero want — no device.

GpuModelMarks

One model’s complete per-model GPU tier state while it is NOT the installed model (a multi-model host keeps one per slot): its routing marks plus the resident driver’s activation. Mint with gpu_model_marks_init; gpu_slot_capture fills it after the model’s load, moe_gpu_model_marks_save/_restore swap it in and out on a switch.

17.1.3. Model loading and sessions

caps(model: Model ): LlmCaps

What model honestly supports at the chat layer (see LlmCaps) — e.g. gemma has no system role, so the chat layer folds the system prompt into the first user turn; system_prompt is false there so callers can surface it instead of being silently absorbed.

Arguments:
create_batch_workspace(model: Model ): BatchWorkspace

Create the caller-owned scratch that eval_batch steps through — one per concurrent batch, reused across calls (buffers grow to the largest batch seen). Holds no session state: the sessions keep their own positions, caches and logits.

Arguments:
create_kv_pool(model: Model; page_rows: int64 = 64; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): KVPool

Create a caller-owned PAGED KV pool over model’s cache geometry. Sessions created over it allocate cache pages of page_rows positions on demand instead of the full seq_len slab up front, so many sessions share one elastic pool. Keep the pool alive as long as sessions live.

Arguments:

17.1.3.1. create_session

create_session(model: Model; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): Session

Create a fresh session (KV cache + scratch) sized to model.config.seq_len — one model, many independent conversations. kv_dtype picks the KV-cache codec: f16 (default, near-lossless) halves bytes, q8_0 halves again (needs head_size/kv_dim % 32 == 0).

Arguments:
create_session(model: Model; pool: KVPool ): Session
load_model(path: string; mode: QuantMode = dasllama_common::QuantMode.fp32 ): Model

Load a model AND its tokenizer from a GGUF file — architecture and tokenizer backend are auto-selected from metadata; mode picks the weight quantization. Q8 loads cache a PREPARED IMAGE beside the gguf for millisecond reloads (DASLLAMA_IMAGE=0 disables); under an active Metal mode the image is the BLOB-ONLY metal flavor (CPU inference against it panics).

Arguments:
release_kv_pages(session: Session )

Return session’s KV pages to its pool (no-op on flat sessions). The normal shape is release + delete; a released session stays alive but loses its cached context — to reuse it, also reset session.n_past to 0.

Arguments:
setup_dasllama_jobque()

Configure the job queue for dasLLAMA’s fork/join matmul dispatch: pooled fork contexts, batched dispatch, and the worker spin-before-park window (jobque_spin_us; 0 disables). Call INSIDE with_job_que(), before the first generate/eval.

17.1.4. Prefix cache

create_prefix_cache(max_groups: int64 = 0 ): PrefixCache

Create a prefix cache for the paged sessions of one create_kv_pool pool: finished streams donate KV pages (prefix_insert), later requests with the same prefix attach them (prefix_attach) instead of re-prefilling. max_groups caps retained groups (0 = unbounded).

Arguments:

  • max_groups : int64

prefix_attach(cache: PrefixCache; pool: KVPool; session: Session; prompt: array<int64> ): int64

Attach the longest cached prefix of prompt to a FRESH paged session of pool: matched pages join the session’s block table and n_past advances past them, so the caller prefills only the tail. Returns the matched count, capped one token short of the prompt.

Arguments:
prefix_chain_list(cache: PrefixCache ): array<PrefixChain>

Snapshot of the cache’s donated chains for dashboards: per donation — page-covered token count, live pages, hit count, born/last-hit ticks, and the caller-provided preview.

Arguments:
prefix_held_groups(cache: PrefixCache ): int64

Pages the cache currently holds (== pool groups retained for reuse).

Arguments:
prefix_insert(cache: PrefixCache; pool: KVPool; session: Session; tokens: array<int64>; preview: string = "" )

Donate a finished session’s KV pages to the cache. tokens is the session’s full EVALED history (only the first n_past rows exist); every full page not already cached survives release_kv_pages. preview labels the chain on the prefix_chain_list surface.

Arguments:
prefix_release(cache: PrefixCache; pool: KVPool )

Release every cached page back to pool and clear the cache (pages still used by live sessions stay alive until those sessions release them). Call before deleting the pool.

Arguments:

17.1.5. Tokenizer

decode(model: Model; ids: array<int64> ): string

Decode a token-id sequence back to text with the model’s tokenizer.

Arguments:

  • model : Model

  • ids : array<int64>

encode(model: Model; text: string; add_special: bool = true; parse_special: bool = false ): array<int64>

Encode text to token ids with the model’s tokenizer. add_special prepends BOS where the model expects one. parse_special is reserved and not yet honored — special tokens reach the model as atomic ids from the chat layer’s template renderer, never by spelling them in text.

Arguments:

  • model : Model

  • text : string

  • add_special : bool

  • parse_special : bool

piece(model: Model; id: int64 ): string

Decode a single token to its text piece — the streaming counterpart of decode.

Arguments:

  • model : Model

  • id : int64

17.1.6. Evaluation and sampling

eval(model: Model; session: Session; tokens: array<int64> )

THE eval primitive: run tokens at the session’s current position and advance it. Prefill = eval(prompt); each generation step = eval([token]) — the same call at different batch sizes. Logits land in session.logits.

Arguments:
eval_batch(model: Model; ws: BatchWorkspace; sessions: array<Session?>; tokens: array<int64> )

One synchronous batched decode step: row i evals tokens[i] at sessions[i]’s current position, advancing each by one — B conversations through ONE pass of the weights (GEMVs batch into GEMMs). Sessions must be distinct, same-geometry, one pool if paged.

Arguments:
eval_embd(model: Model; session: Session; embd: array<float>; npos: int64; non_causal: bool = false )

eval’s embedding-input twin: prefill npos pre-built embedding rows (npos × dim, token-major) at the session’s current position and advance it — the multimodal splice entry. non_causal marks the whole call as one image span (gemma vision); audio rows stay causal.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • non_causal : bool

eval_embd_span(model: Model; session: Session; embd: array<float>; npos: int64; span_lo: int64; span_hi: int64 )

eval_embd for a prompt carrying one NON-CAUSAL image span: rows [span_lo, span_hi) prefill with every query attending the whole span, the text around them causally — the gemma vision decode shape (mtmd’s non-causal media chunk).

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • span_lo : int64

  • span_hi : int64

eval_embd_span_mrope(model: Model; session: Session; embd: array<float>; npos: int64; span_lo: int64; span_hi: int64; grid: int2 )

eval_embd_span’s qwen mrope twin: the span rows rope as a grid-shaped merged image (position advance max(grid.x, grid.y), tracked on the session for every later eval); same span mask, only the angles differ. Panics without rope.dimension_sections.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • span_lo : int64

  • span_hi : int64

  • grid : int2

sample(session: Session; params: SamplingParams ): int64

Sample the next token from session.logits per params: penalties, then temperature/top-k/top-p/min-p and a CDF draw — or greedy argmax when params.temp <= 0 (SamplingParams() defaults are greedy).

Arguments:
set_seed(session: Session; seed: int )

Seed the session’s sampling RNG for reproducible generation.

Arguments:
stats(session: Session ): Stats

Timing of the most recent generate/respond call on session: prompt/generated token counts, time to first token, prefill and generation tok/s.

Arguments:

17.1.7. Generation

generate(model: Model; session: Session; prompt: array<int64>; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

Stream-generate up to max_tokens from prompt, invoking the trailing block per token with (id, piece); return false from the block to stop early. Prefills the prompt in one eval, then samples one token at a time; returns the number of tokens emitted.

Arguments:

  • model : Model

  • session : Session

  • prompt : array<int64>

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

17.1.7.1. generate_embd

generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

generate’s embedding-prefill twin: prefill npos pre-built embedding rows (the multimodal splice — see eval_embd), then stream-sample exactly like generate. The chat layer’s audio turns run on this; use it directly for custom multimodal prompts.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; span_lo: int64; span_hi: int64; blk: block<(id:int64;piece:string):bool> ): int64
generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; span_lo: int64; span_hi: int64; grid: int2; blk: block<(id:int64;piece:string):bool> ): int64

17.1.8. Embeddings

embed(model: Model; text: string ): array<float>

Mean-pooled, L2-normalized sentence embedding of text (model.config.dim floats): the decoder’s last-layer hidden state (post-final RMSNorm), averaged then unit-normalized. A decoder-only model used this way yields RAG-grade vectors, not a dedicated embedder’s.

Arguments:

  • model : Model

  • text : string

17.1.9. Vision and audio encoders

encode_audio(embedder: AudioEmbedder; scratch: AudioState; samples: array<float>|array<float>#; out: array<float> ): int64

encode_image’s audio twin: 16 kHz mono samples through the carried family’s encoder into out = npos × proj_dim soft tokens (returns npos). A scheduler owning its own embedder calls this; the server’s media worker is the worked example.

Arguments:
encode_image(embedder: VisionEmbedder; scratch: VisionState; img: VisionImage; tag: string; out: array<float> ): int64

The whole image path in one call — geometry, letterbox, normalize, encode — into out = npos × proj_dim soft tokens (returns npos). add_user_image is this plus queueing onto a chat turn; a scheduler owning its own embedder calls this. tag names the vision dump.

Arguments:

17.1.10. Chat

add_assistant(model: Model; chat: ChatSession; text: string )

Inject a KNOWN assistant reply (no generation): prefill the pending user turn and text into the KV cache, then close the turn — like respond but with a supplied reply. The shape a stateless server needs to replay history. Precondition: a user message is pending.

Arguments:
add_user(chat: ChatSession; text: string )

Queue a user message for the next respond.

Arguments:
add_user_audio(chat: ChatSession; samples: array<float>|array<float># ): auto

Queue audio (16 kHz mono f32 PCM) for the next respond — encoded to soft tokens immediately and spliced at the head of the turn before any add_user text. Needs a chat created with create_chat(model, tower); call inside with_job_que().

Arguments:

  • chat : ChatSession

  • samples : option<array<float>| array<float>#>

add_user_image(chat: ChatSession; img: VisionImage )

Queue an image for the next respond — geometry, letterbox and the embedder run NOW, spliced at the head of the next user turn. ONE image per turn; needs a chat created with create_chat(model, embedder); call inside with_job_que() (decode: dasllama_vision_io).

Arguments:

17.1.10.1. create_chat

create_chat(model: Model; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): ChatSession

Start a conversation over model: resolves the chat template (GGUF-embedded, falling back to the arch registry) and creates the session. system is the system prompt (empty = none); max_new caps each reply; kv_dtype is the session’s KV-cache codec.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

  • kv_dtype : KVDtype

create_chat(model: Model; embedder: VisionEmbedder; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): ChatSession
create_chat(model: Model; tower: AudioTower; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): ChatSession
create_chat_renderer(model: Model; system: string = ""; max_new: int64 = 256 ): ChatSession

create_chat’s RENDER-ONLY twin: resolves the template/stop ids/turn close but creates NO KV session — a queued request can render its whole prompt holding tokens only, no cache memory. It cannot respond/eval.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

render_assistant(model: Model; chat: ChatSession; text: string; out: array<int64> )

add_assistant’s render half: appends the exact token stream a known reply prefills to out WITHOUT running the model, advancing the transcript like add_assistant. Use on a create_chat_renderer chat to replay history with no KV memory. Precondition: user message pending.

Arguments:
render_close(model: Model; chat: ChatSession ): array<int64>

The tokens that TERMINATE an assistant turn (what respond evals after the reply) — for schedulers that close a finished stream’s turn themselves.

Arguments:
render_turn(model: Model; chat: ChatSession ): array<int64>

Render the next turn’s prefill token ids — BOS + system on the first turn, then the user turn and the generation prompt — WITHOUT running the model. For inspection, token budgeting, tests.

Arguments:
render_turn_audio(model: Model; chat: ChatSession; head: array<int64>; tail: array<int64> )

render_turn’s AUDIO twin: the same two-span contract around the audio soft-token splice (the template’s audio span markers). Render-only (no KV, no encoder): the rows come from encode_audio, and a scheduler prefills head, rows, tail.

Arguments:
render_turn_image(model: Model; chat: ChatSession; head: array<int64>; tail: array<int64> )

render_turn’s IMAGE twin: the turn’s prefill as the two token spans that bracket the image soft-token splice — head before the rows, tail after. Render-only (no KV, no embedder): the rows come from encode_image, and a scheduler prefills head, rows, tail.

Arguments:
respond(model: Model; chat: ChatSession; params: SamplingParams; blk: block<(piece:string):bool> ): string

Generate the assistant’s reply to the queued user message, streaming pieces through the trailing block (return false to stop early). Terminates the turn in the KV cache and appends both turns to chat.history; returns the full reply text.

Arguments:
set_thinking(chat: ChatSession; on: bool )

Toggle reasoning for a hybrid thinking model (Qwen3 family): false appends the template’s empty think block so the model answers directly. No-op without a suppress form or think specials in the vocab; default is on.

Arguments:

17.1.11. Tool calling

17.1.11.1. add_tool_results

add_tool_results(chat: ChatSession; results: array<string> )

Queue tool results as the next pending turn — the reply to an assistant turn that called tools. Call in place of add_user, then respond/render_turn as usual.

Arguments:
add_tool_results(chat: ChatSession; results: array<string>; names: array<string> )
parse_calls(chat: ChatSession; reply: string ): ToolReply

Parse a complete reply per the model family’s wire format into a ToolReply — the reasoning span, the content, and the calls with arguments normalized to JSON object text. A family with no tool format returns the reasoning/content split alone (safe on every reply); the buffered twin of the server’s streaming parse.

Arguments:
render_assistant_calls(model: Model; chat: ChatSession; text: string; calls: array<string>; out: array<int64> )

render_assistant’s tool-calling twin: replay an assistant turn that emitted tool calls (verbatim \{"name":…,"arguments":…} objects) plus any text alongside.

Arguments:

  • model : Model

  • chat : ChatSession

  • text : string

  • calls : array<string>

  • out : array<int64>

set_tools(chat: ChatSession; tools: array<string> )

Declare the conversation’s tools (verbatim OpenAI tools[] JSON objects, moved in) BEFORE the first turn renders — the system turn carries the family’s tool block. Families with no tool format (tmpl.tool_call_open empty) ignore them.

Arguments:

17.1.12. Reasoning (thinking models)

effective_stop_ids(chat: ChatSession ): array<int64>

The stop ids in force for the NEXT generation: the template’s stops plus its thinking-off extras while thinking is off. Schedulers that stop streams themselves read this, not chat.stop_ids, so an instruct-mode gemma-4 cuts at a stray channel marker.

Arguments:
make_think_stream(chat: ChatSession ): ThinkStream

The incremental reasoning/content splitter for chat’s next turn — feed streamed pieces through think_feed, flush with think_finish. Armed only when the turn actually thinks (toggle on, markers in the vocab, gate rendered) — else a pass-through stream.

Arguments:
split_reasoning(chat: ChatSession; reply: string ): ThinkSplit

Split a complete reply at its reasoning boundary per the model family’s reply format (<think> pair, Harmony channels, gemma-4’s thought channel). Both halves come back stripped when a reasoning span is found; a reply with no reasoning passes through untouched.

Arguments:
think_drain(ts: ThinkStream; full: string ): ThinkSplit

Drain a COMPLETE reply through the splitter in one call: feed + finish + the strip rule (both halves strip when a reasoning span was consumed). The buffered-response twin of the think_feed/think_finish streaming pair.

Arguments:
think_feed(ts: ThinkStream; piece: string; reasoning: string&; content: string& )

Feed one streamed piece through the splitter; the out-strings are OVERWRITTEN with this piece’s reasoning/content deltas (either may be empty while a partial marker is held).

Arguments:

  • ts : ThinkStream

  • piece : string

  • reasoning : string&

  • content : string&

think_finish(ts: ThinkStream; reasoning: string&; content: string& )

Flush the splitter at end-of-generation (OVERWRITES the out-strings with the final deltas): an unclosed reasoning span classifies as reasoning — the truncated-tail rule.

Arguments:

  • ts : ThinkStream

  • reasoning : string&

  • content : string&

17.1.13. Operations: prepared images and dispatch

dlim_clean(gguf_path: string; apply: bool; keep_other: bool = false ): tuple<total:int;stale:int;removed:int;freed:int64>

Garbage-collect STALE and OTHER images beside gguf_path (FOREIGN, another flavor’s, are always left alone): apply = false only reports, true removes; keep_other spares OTHER. Returns (total, stale, removed, freed bytes).

Arguments:

  • gguf_path : string

  • apply : bool

  • keep_other : bool

dlim_inventory(gguf_path: string ): array<DlimImageInfo>

List the prepared images (.dlim) minted beside gguf_path — per image: file, bytes, image version, identity, and a verdict (CURRENT loads; STALE vN is an older image version; OTHER a different bake configuration or box; FOREIGN a different flavor).

Arguments:

  • gguf_path : string

get_dispatch_worker_limit(): int

The dispatch worker cap in force (0 = no limit) — set_dispatch_worker_limit’s read half.

kernel_backend_available(name: string ): bool

True when kernel backend name is registered AND its availability witness passes on this box — the detection probe behind defaults-first backend selection (the vulkan witness enumerates devices once and caches its verdict). Unknown names are false.

Arguments:

  • name : string

select_matmul_backend_for_load(): bool

Select the best matmul backend for the NEXT model load (honors a pin). Returns true when the choice needs the loader to repack weights into its interleaved layout.

set_dispatch_worker_limit(v: int )

Cap the kernel-dispatch worker count (0 = no limit, all job-que workers). Latched by setup_dasllama_jobque — set it before that call.

Arguments:

  • v : int

17.1.14. Operations: GPU tier and model slots

gpu_model_marks_init(): GpuModelMarks

A fresh no-model marks holder — one per model slot; the marks save/restore pair swaps tier state across switches. (A generated constructor is not callable through the facade typedef, so the holder is minted here.)

gpu_slot_capture(st: GpuModelMarks ): string

Capture the INSTALLED model’s tier state into st right after its load_model and classify it — “gpu:resident” | “gpu:rails” | “cpu” — from the CAPTURED marks, never the process-global VRAM counter (it still holds the previous load’s bytes on a later CPU load).

Arguments:
gpu_slot_rearm(want: GpuTierWant; t: Model ): string

Re-arm the INSTALLED model onto the GPU tier — want, arm, resident upload (bake-slice path for a mapped vulkan-flavor t). Its marks are installed and the device holds no other model (moe_gpu_drop_model first). Returns “gpu:resident” | “gpu:rails” | “cpu”.

Arguments:
gpu_tier_status(): GpuTierStatus

Snapshot of the GPU tier in force: VRAM budget, resident/streamed layer counts, dense planes, engage/decline state — the server’s GPU badge reads this.

moe_gpu_drop_model()

Drop the INSTALLED model’s whole device state (resident stacks, mirrors, VRAM) — the evict half of a slot switch. Caller guarantees quiescence: no step in flight, every live session hydrated first.

moe_gpu_hydrate_session(t: Model; s: Session )

Pull s’s host KV back from the device mirror when it owns live mirror rows (no-op otherwise). Run over every live session of the outgoing model BEFORE moe_gpu_drop_model — the drop destroys the mirror.

Arguments:
moe_gpu_model_marks_restore(st: GpuModelMarks )

Install st as the per-model GPU tier state — moe_gpu_model_marks_save’s inverse; st reads as no-model after.

Arguments:
moe_gpu_model_marks_save(st: GpuModelMarks )

Save the INSTALLED model’s GPU tier state into st and disarm it — one half of the multi-model slot switch (the engine’s tier state is per-process, not per-model).

Arguments:
moe_gpu_tier_arm(): bool

Arm the recorded tier want so a GPU backend can install its hooks — call between set_gpu_tier_want and load_model. False when no backend claims the request.

moe_gpu_weight_budget(): int64

The armed backend’s resident-weight VRAM budget in bytes (0 = no GPU backend armed). CAVEAT: the hook device-inits on first call — a reporting caller gates on gpu_tier_status().device being non-empty first (supported must not be the gate).

set_gpu_tier_want(want: GpuTierWant )

Record the GPU tier request the NEXT moe_gpu_tier_arm/load_model honors — the programmatic form of the core DASLLAMA_GPU_* knobs (an env var present overrides its field; the classifier/dense-arm knobs are env-only). A zero want keeps the load off the device.

Arguments:
set_resident_prefill_allowed(allowed: bool )

Allow or pin out the resident-decode prefill arm. A multi-stream scheduler pins it OFF for good: the single shared mirror plus chunked prefill would leave device-only KV that a second stream’s steal strands.

Arguments:

  • allowed : bool