Curriculum step 1 — the dominant LLM kernel, naive and correct first. matmul(y, w, x, n, d): y = W·x, W row-major [d×n], matching llama2.c's layout so intermediate activations cross-check directly. Baseline (M1 Max, single thread): naive JIT ~3.7 GFLOP/s — already at llama2.c -O3 single-thread; interpreter ~0.32 GFLOP/s (~12× behind). ~3.5% of fp32 peak; SIMD/blocking/threading runway is next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RMSNorm — root-mean-square norm (no mean subtraction, no bias), the stabilizer before each matmul block. o[i] = weight[i] * x[i] / sqrt(mean(x^2) + eps), eps = 1e-5, matching llama2.c. Tests assert the defining properties (scale removed → constant vector maps to ones; unit weights → mean(o^2) == 1; normalized input passes the learned weight through) rather than bit values, since sqrt+eps makes exact hand-computation noisy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
softmax(var x, size) — in-place, turns attention scores into a probability distribution. Subtracts the row max before exp() so large scores can't overflow (exp(x-max)/sum is identical math, numerically safe), matching llama2.c. Tests: sums to 1 + order-preserving, uniform input -> uniform output, and shift-invariance/overflow-safety (softmax([1000,1001,1002]) == softmax([0,1,2]), no NaN). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
silu(var x, size) — in-place SiLU/swish activation, x/(1+exp(-x)), the nonlinearity in the SwiGLU FFN gate. Matches llama2.c. Tests: hand-computed values (silu(1)=0.731, silu(2)=1.762, silu(-1) negative — non-monotonic dip) and saturation (silu(x)->x for large +x, ->0 for large -x). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rope(var vec, pos, head_size, n) — rotary position embedding, in place. Rotates each adjacent pair (2j, 2j+1) within a head by pos/10000^(2j/ head_size), encoding position as rotation. Matches llama2.c's adjacent-pairs scheme. Called once for Q (n=dim), once for K (n=kv_dim). Tests: pos 0 = identity, rotation preserves pair norm, known angle ([1,0] at pos 1 -> [cos 1, sin 1]), and the defining relative-position invariance (Q.K depends only on the position offset). This completes the step-1 math primitives: matmul, rmsnorm, softmax, silu, rope — all naive fp32, JIT-tested. Next: wire the forward pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prep for the forward pass. Every array param on all 5 primitives is now `array<float> | #` — accepts an owning array OR a borrowed view (array<float>#), so weights can be passed zero-copy as views into one big model blob (via daslib/array_boost array_view) and outputs can be written straight into a view (e.g. the KV cache). The OR-type monomorphizes, so no runtime dispatch. Dims/indices switched to int64 (range64) for large models, matching llama2.c's unsigned-long-long offset math. Profiled int64 vs int32 matmul: within noise on M1 Max (classifier 5.06 vs 5.15 ms) — int64 is free here. Baseline holds: proj 46us, classifier 5.1ms in JIT. Tests updated to int64 literals; all 20 subtests green, lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
load_checkpoint(path) reads a llama2.c checkpoint via fmap (binary-safe mmap), parses the 7-int32 header, and copies the fp32 weights into one owned array<float> blob. Weight tensors are addressed by int64 base offsets into the blob (float-element units), a transliteration of llama2.c's memory_map_weights — the forward pass will array_view into the blob (zero copy) rather than splitting per-layer arrays. Verified on stories15M: Config matches (dim=288, 6 layers, 6 heads, vocab=32000, head_size=48, kv_dim=288, shared_weights), blob=15,204,000 floats == file size, and the offset table exactly tiles the blob (rms_final + dim + skipped freq_cis region = blob end). dasllama_run.das is the local checkpoint runner (model is 60MB, gitignored — runs locally, not CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
forward(t, s, token, pos): faithful port of llama2.c forward() built from the 5 primitives. Weights are zero-copy views into the blob (array_view via mm_at/rms_at helpers); per-head attention (scaled Q·K -> softmax -> weighted sum of V) is written inline; KV cache grows by position. generate() greedy-decodes from BOS. RunState holds the scratch buffers + flat KV cache, allocated once. MILESTONE A: dasllama_run.das greedy-decodes stories15M from BOS and matches llama2.c (empty prompt, -t 0.0) token-for-token across 12 tokens [9038, 2501, 263, 931, 29892, 727, 471, 263, 2217, 7826, 4257, 365] = "Once upon a time, there was a little girl named L...". Holds under both interpreter and JIT. The naive fp32 forward pass is correct. (Local validation only — stories15M.bin is 60MB, gitignored.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resize and range64 both take int64 directly, so the int(c.dim) / range(int(vocab)) downcasts were defeating the int64 conversion. make_run_state now resizes with int64 dims; argmax loops range64 and returns int64; generate uses range64 and keeps token ids int64 end-to-end, removing the int64(pos)/int64(next) casts. Token-id lists (generate result, ORACLE) are now array<int64>. Only remaining conversions are int64(hdr[i]) in the loader — the necessary int32-file-header to int64-Config widening. Still matches llama2.c token-for-token (interp + JIT). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dasllama_tokenizer.das: load_tokenizer parses tokenizer.bin (int32
max_token_length, then per token {float32 score, int32 len, len bytes})
via fmap; each piece is string(array<uint8>) over the byte range.
decode(prev, token) ports llama2.c decode — strips a single leading
space right after BOS (slice) and turns raw-byte tokens "<0xXX>" into
the actual byte (peek_data + hex parse).
Runner now prints the decoded story:
"Once upon a time, there was a little girl named L"
still matching the oracle token-for-token (interp + JIT).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
encode(tk, text, bos, eos) ports llama2.c encode: optional BOS, a
dummy leading-space token, per-UTF8-codepoint vocab lookup (byte
fallback = byte+3), then the greedy highest-score adjacent-pair BPE
merge. Backed by a piece->id table built at load.
generate now takes prompt_tokens: it forces tokens while inside the
prompt, then samples (argmax) — so real prompts work.
Validated on stories15M:
- encode("Once upon a time") == [1, 9038, 2501, 263, 931] (matches
llama2.c tokenization)
- empty-prompt greedy still matches the oracle token-for-token
- prompt-driven generation reproduces the oracle story: "Once upon a
time, there was a little girl named Lily. She loved to play outside
in the sunshine. One day, she saw a big, red ball in the sky..."
The tokenizer (encode + decode) is complete; real text in, real text out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dasllama_perf.das times a greedy decode of N tokens and reports tok/s + matmul GFLOP/s — the real-world metric for the perf work, so each optimization is judged on throughput, not just micro-benchmarks. Baseline (M1 Max, JIT, stories15M): 108.5 tok/s, ~3.3 GFLOP/s — already at llama2.c -O3 single-thread parity. That's the starting line for the optimization ladder (SIMD -> bounds elision -> threading). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Experiment ledger: matmul_variants.das keeps every kernel attempt (matmul_naive, matmul_f4) so bench_matmul.das can compare them head-to-head with a correctness gate; the winner is promoted into the production dasllama_math::matmul. float4 accumulates the dot product four lanes at a time (*reinterpret<float4?> loads) and horizontal-sums, with a scalar tail. Kernel: ~3.4x over naive (proj 46->13us, classifier 5.06->1.50ms; ~3.7 -> ~12.4 GFLOP/s). Promoted to production. End-to-end (stories15M, JIT): 108.5 -> 301.4 tok/s (2.78x) — now past llama2.c -O3 single-thread. The 3.4x kernel / 2.78x e2e gap is Amdahl: the still-scalar attention/rmsnorm/softmax now dominate more. Still matches llama2.c token-for-token (float4 reorder doesn't flip argmax). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
matmul_llvm_vec (Boris): a per-row dot_vec the LLVM JIT auto-vectorizes
via [hint(unsafe_range_check, noalias)] + for[vectorize,
vectorize_width=8]. Beats the hand-rolled float4 by ~1.8x — width-8 +
FMA + multiple accumulators + bounds-elision, all from the JIT given
the right hints.
Ledger (bench_matmul, JIT, correctness-gated):
naive f4(4-wide) llvm(8-wide)
proj 288x288 46.1us 13.1us 7.1us
classifier 4.95ms 1.46ms 0.79ms (~3.7 / 12.4 / 23 GFLOP/s)
Promoted into production dasllama_math::matmul (dot_vec helper + row
loop; reinterpret normalizes the weight-view `#` pointer to a plain
pointer dot_vec accepts). End-to-end stories15M JIT: 108.5 -> 573.7
tok/s (5.3x over naive, 5.3x over llama2.c single-thread). Still
token-for-token with llama2.c.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swept vectorize_width x unroll_count in the ledger (classifier, JIT):
w8 (no unroll) 811k ns
w8 unroll 2 745k <- best, stable
w8 unroll 4/8 ~750k
w16 776k (no gain — M1 NEON is 128-bit/4-wide, so width
16 just lowers to more ops unroll already does)
w16 unroll 4 753k
unroll_count=2 promoted into dasllama_math::dot_vec (~8.5% kernel).
End-to-end: 573.7 -> ~594 tok/s (Amdahl dilutes the kernel gain).
Still token-for-token with llama2.c. Variants kept in the ledger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promoted the matmul inner kernel to a public dot() primitive and added axpy() (d += scale*s), both LLVM auto-vectorized (width-8, unroll-2, noalias, bounds-elided). Rewired attention: Q·K score = dot(), the value weighted-sum = axpy() per cached position. matmul now uses dot(). End-to-end ~594 -> ~640 tok/s (~8%), bigger than the FLOP estimate predicted — the scalar attention loops carried bounds-check + inner- loop overhead beyond raw FLOPs. Still token-for-token with llama2.c. Now ~5.9x over llama2.c single-thread. Single-thread story largely done. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Threading the matmul output rows across worker threads (parallel_for). Workers read the main-context y/w/x through raw pointers (unsafe cross-context, as designed) and run dot() per row. Granularity, settled by sweep (bench_matmul_par.das, classifier shape): - num_jobs = N (one fat chunk per worker). 2N ties; 10N+ regress on per-job overhead; "each dot"/"each 10 dots" are 30-300x SLOWER (thousands of job allocations). - parallel_for has a ~90us fixed cost, so only matmuls with n*d above ~2M win: classifier 32000x288 (9.2M) -> 2x; ffn 768x288 (0.22M) and proj 288x288 would be 5-13x SLOWER. matmul auto-dispatches on MATMUL_PAR_THRESHOLD; small matmuls + unit tests stay single-thread (need no job queue). Generalizes: real 1B+ models clear the threshold on every layer matmul, so threading pays off far more there. - The classifier caps at ~2x = memory-bandwidth bound (37MB streamed); cores can't beat the bus, only quantization can. with_job_que wraps main (persistent pool, no per-token create). matmul re-exports jobque_boost (parallel_for macro expands _::notify symbols in the consumer scope). matmul_par + the num_jobs sweep kept in the ledger. End-to-end stories15M JIT: ~640 -> ~839 tok/s (~7.8x over llama2.c single-thread). Still token-for-token (independent rows = identical). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The unit tests only used tiny shapes (single-thread path); the threaded path was exercised only by the manual dasllama_run (needs the 60MB model). test_matmul_par runs production matmul on a 288x8192 shape (2.36M >= threshold, so it threads — and must be inside with_job_que) and compares against the scalar matmul_naive ground truth. A threading bug (bad partition, race, pointer) yields large errors; fp reorder stays under 1e-2. Model-free, runs anywhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_forward asserts the full pipeline (BPE encode + forward + greedy decode) matches llama2.c token-for-token — the oracle that was only checked by the manual dasllama_run. Gated on stat(stories15M.bin): runs the assertions locally where the model is present, skips cleanly in CI (model is 60MB, gitignored). Threads the classifier (with_job_que). Whole module now sweeps green via one command: daslang dastest/dastest.das -- --test modules/dasLLAMA/ (24 subtests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Quantization arc, position 0 (detector before kernel). Q8_0 format (32 weights -> int8 + one fp32 scale) plus a detector that measures the loss before any fast kernel exists: - reconstruction error (per-weight, SNR in dB) - divergence step (first token Q8 greedy leaves fp32) - coherence (both stories decoded for eyeball) The trick: dequantize the Q8 weights and run the EXISTING fp32 forward on them — no new kernel needed to measure quality. On stories15M: SNR 44.8 dB, 3.56x smaller, Q8 matches fp32 token-for-token over 64 tokens (effectively lossless at greedy decode). 6 round-trip unit tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ledger experiment for the Q8 matmul kernel (int8 weights + per-block fp32
scale x fp32 activations). Findings on the stories15M classifier (32000x288):
- Single-thread, Q8 LOSES to fp32 (q8f4 1037us vs fp32 742us). fp32 is
already bandwidth-bound at ~49 GB/s (single-core ceiling), so there are
no bytes to reclaim — the int8->float conversion is pure added compute.
- Threaded, Q8 WINS: q8f4 par-2N 213us vs fp32 par-N 406us = 1.91x. Q8
trades a bandwidth-bound problem for a compute-bound one, and on M1
compute threads ~5-7.5x while bandwidth plateaus at ~2.1x. par-2N beats
par-N (E-core work-stealing), the opposite of fp32's plateau.
Variants tried (all correctness-gated vs fp32-on-dequantized-weights):
dot_q8 (per-block reduce) 1752us; q8b (dequant-to-scratch + dot_vec)
1118us; dot_q8_f4 (float4 acc, one reduce/row) 1037us = winner.
matmul_q8f4_par is the threaded promotion candidate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…till wins)
Inspected the JIT LLVM IR (daslang -jit file.das -- --jit-dump) for the dot
kernels:
- float(int8) compiles straight to `sitofp <8 x i8> to <8 x float>` — the
int8->float widen IS vectorized, and the inner int() was redundant. Dropped
it in dot_q8 / dot_q8_f4 / dequant_row / dequantize_q8_0.
- The conversion is NOT the bottleneck. dot_q8's cost is a per-block
`llvm.vector.reduce.fadd` (runs nb times/row) forced by the per-block
scalar scale; fp32 dot reduces once/row with loop-carried accumulators.
Added dot_q8_flat (flat row loop, scale via ws[k/32]): the JIT broadcasts the
scale across each unrolled 32-chunk and keeps loop-carried vector accumulators
(one reduce/row) — clean IR, but it trades the per-block reduce for a
per-element scale multiply (2 fmuls/elem) and loses: ST 1253us, threaded
par-2N 243us. dot_q8_f4 stays the winner (ST 1038us; threaded par-2N 219us =
2.1x over fp32 par-N 467us). Kept flat in the ledger as a recorded result.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…en [test] Promoted the winning Q8 kernel (dot_q8 float4 + auto-dispatching matmul_q8, par-2N for large) into dasllama_math. Transformer gains a quantized copy of the whole blob (qblob:int8 + qscales, parallel layout so the same offsets index it) plus a use_q8 toggle; quantize_weights() fills it. forward routes all eight 2D weight matmuls (Q/K/V/O, w1/w2/w3, classifier) through Q8 when use_q8; norms and the embedding lookup stay fp32 (matches how GGUF keeps 1D norms full-precision). Results on stories15M (perf harness A/B, same loaded model): fp32 887 tok/s -> all-weights Q8 981 tok/s = 1.1x, token-for-token (test). Measured both scopes: classifier-only is actually faster+more-accurate on this toy (1.18x, exact to 221 tok) because the per-layer matmuls are L2-resident / compute-bound single-thread (Q8 regresses them) and all-weights accumulates quant error through the residual stream. But all-weights is right for real models (every matmul large+bw-bound; memory). Decision: keep all-weights; tune the scope (the future quant config) on a real 7B model. dasllama_quant_eval now exercises the real Q8 forward for its divergence check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GGML-exact Q4_0 layout: 32-weight blocks, split-half nibble packing (byte j = elem j low / j+16 high), offset-8, d = smax/-8; fp32 scale in memory (GGUF fp16 transcodes at load). quantize_q4_0/dequantize_q4_0 + 5 round-trip tests + a position-0 detector (dasllama_quant_eval_q4). On stories15M: SNR 20.9 dB, 6.4x smaller, greedy diverges at token 19 but stays fully coherent. The dequant->fp32-forward output is the correctness gate for the upcoming dequant-in-dot kernel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…[test] Q4 dequant-in-dot kernel: per-block float-vectorized nibble unpack at vectorize_width=4 (M1 native NEON lane). Hint sweep showed width=4 beats width=8 by ~7% — the heavy Q4 unpack (mask+shift+sub+sitofp per element) hates 8-wide cross-lane splitting, opposite of Q8's lighter unpack which liked width=8. IR confirmed: w4u4 emits pure <4 x float>, no shuffles. Flat (single loop) variant dies on Q4: split-half packing makes the index non-affine -> scalar-gather storm. while==for inner (byte-identical IR), vectorize_predicate a NEON no-op. Promoted dot_q4/matmul_q4 into dasllama_math (auto-dispatch, threaded 2N). Wired all-weights Q4 into forward via a QuantMode enum (fp32/q8/q4) + q4blob/q4scales + quantize_weights_q4 + mm_at_q4. test_forward_q4 (oracle prefix matches), perf gains a Q4 leg, detector runs the real Q4 forward. Toy results (stories15M, M1 Max): Q4 723 tok/s = 0.87x fp32 (NET REGRESSION — heavier unpack hurts the L2-resident per-layer matmuls more than the threaded classifier wins; same mechanism as all-weights Q8, amplified). Real win awaits the bandwidth-bound GGUF model. SNR 20.9 dB, 6.4x smaller, greedy diverges at token 21, fully coherent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…f [step 7.0] Generic GGUF reader (dasllama_gguf): header + typed KV metadata + tensor-info table out of an mmap'd buffer, with typed getters (int/f32/str + str/f32/i32 arrays) and f16_to_f32. Mirrors llama.cpp gguf_context / llama_model_loader split. Validated (dasllama_gguf_dump, position-0 detector) against llama-gguf on both the small llama-spm vocab file and TinyLlama-1.1B-v0.3 F16: data_offset and every tensor offset match byte-exactly. TinyLlama: arch=llama, dim=2048, 22 layers, n_heads=32/n_kv_heads=4 (GQA kv_mul=8), vocab=32003 (no vocab_size key -> derive from token_embd), rope_theta=10000, rms_eps=1e-5 (match our primitives).
… llama.cpp [step 7.1]
load_gguf maps a llama-arch GGUF (F16/F32) into the existing memory_map_weights
blob layout, so the proven fp32 forward + KV cache run unchanged. Config from
metadata (dim/hidden/layers/heads); vocab derived from token_embd (no vocab_size
key); classifier untied (separate output.weight). ggml's row-major [out x in]
weight layout == our llama2.c convention, so no transpose.
MILESTONE B: first REAL model. TinyLlama-1.1B-v0.3 F16 reproduces instrumented
llama-simple greedy 40/40 token-for-token ('Once upon a time, there was a young
girl named Lily...'). GQA now actually exercised (kv_mul=8, was 1 on the toy).
~6s wall incl JIT codegen + 2.2GB mmap + 1.1B-elem F16 decode + 45 forwards.
dasllama_gguf_run = the loader check (oracle = instrumented llama-simple, prompt
IDs from llama-tokenize so only loader+forward are under test).