7.17. dasllama-server — an OpenAI-compatible server over dasLLAMA
dasllama-server (directory: utils/dasllama-server/) is a drop-in
OpenAI-compatible HTTP server for dasLLAMA
CPU inference, written entirely in daslang over the public dasllama
facade plus the dasHV HTTP layer. Point any
OpenAI client (opencode, Open WebUI, the llm CLI, the openai Python SDK,
…) at http://127.0.0.1:<port>/v1.
It reaches only public facade verbs — load_model / create_chat /
add_user / add_assistant / respond / transcribe / embed —
and that is the point: the server is the acceptance test for the dasLLAMA API
rework. If a full OpenAI surface builds with no reach into engine internals, the
facade is complete.
7.17.1. Run
Run under -jit — interpreted inference is far too slow for model work:
bin/daslang -jit utils/dasllama-server/main.das -- --model <model.gguf> \
[--port 8080] [--quant q8] [--asr <asr.bin>] [--mmproj <mmproj.gguf>] \
[--image-mmproj <mmproj.gguf>] [--ctx 4096] [--tune]
Flag |
Short |
Default |
Meaning |
|---|---|---|---|
|
|
(required) |
GGUF model to serve |
|
|
|
Listen port |
|
|
|
Weight quantization: |
|
|
— |
ASR model (whisper / parakeet / qwen3-asr) — enables the |
|
— |
mmproj GGUF for the Qwen3-ASR route (paired with |
|
|
— |
vision mmproj GGUF (gemma-4 family) — the chat route then accepts
|
|
|
|
Context-length cap in tokens |
|
|
— |
Re-tune this box’s dasLLAMA kernels, then relaunch (see Per-box tuning) |
|
|
|
— |
Show help and exit |
Generation runs on one tick thread: a scheduler interleaves every live stream’s decode, so concurrent requests share the model rather than queue behind each other. Audio transcription and image encoding each run on their own worker thread and rejoin the tick loop with their result. OpenAI is stateless — the client resends the full transcript each turn.
7.17.2. Per-box tuning
The server declares [tune_policy(missing = "auto")], so the first start
on an untuned box runs the dasLLAMA kernel tuner (gen_tune_probe), writes
the per-box manifest, and relaunches itself with the winners; thereafter it
serves directly and logs the tune status at startup. --tune forces a
re-tune. DAS_TUNE_POLICY=error skips per-start tuning while developing (it
prints the tuner command instead of running it).
The winners live at <das_root>/dasllama.tune.json and are shared by every
dasLLAMA application on the box — no per-app scope declaration, since requiring
dasLLAMA pulls in its [tune_scope]. Two sibling CLI tools ship alongside
the server, each with the same [tune_policy(missing = "auto")] and reading
the same manifest: ask (a one-shot --prompt → completion, reporting
ttft and prefill/decode t/s) and wav2txt (an --file audio → transcript,
reporting decode/transcribe time and the real-time factor). Whichever of the
three you run first tunes the box; the rest are then instant. See
Kernel tuning for the framework, and
modules/dasLLAMA/tune_for_this_box.md for the measurement discipline.
7.17.3. Endpoints
Method |
Path |
Notes |
|---|---|---|
|
|
Lists the served model (and |
|
|
Chat; |
|
|
Raw completion; |
|
|
Mean-pooled, L2-normalized sentence embeddings |
|
|
Speech→text (multipart upload; needs |
|
|
Speech→English text (needs |
|
|
|
|
|
|
|
|
|
|
|
Scheduler counters (media ones include |
|
|
Per-stream states + text tails, prefix-cache chains, recent ASR jobs |
|
|
Per-slot prepared-image ( |
|
|
Silero speech spans over an uploaded clip — the control page’s waveform overlay (≤120 s) |
|
|
The curated model list with local presence and per-entry download state |
|
|
|
|
|
Read bench state and log / start the quiesced A/B benchmark against the configured llama.cpp binary ( |
|
|
Read bake state and log / bake the slot’s prepared |
|
|
Effective config with per-key source / validate and save an authoritative TOML, applied on the next restart ( |
|
|
The tune-sidecar exchange surface: policy plus the current sidecar’s identity and share state |
|
|
Live lookup of this box against the exchange (a network call — seconds) |
|
|
|
|
|
Privacy-strip and submit this box’s own tune to the exchange |
|
|
Arm a local re-tune and restart — the next boot races this box |
|
|
Schedule a validated collection at the next lifecycle safe point; concurrent requests coalesce |
|
|
Drain, then exit 4 — the watchdog relaunches with the saved config |
|
|
Graceful stop |
7.17.3.1. Chat
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "Say hello in one word."}],
"max_tokens": 16, "stream": false
}'
7.17.3.2. Embeddings
input is a string or an array of strings. Each vector is model.config.dim
floats, mean-pooled over the decoder’s last-layer hidden state (post-final-norm)
and L2-normalized. A decoder-only LLM used as an embedder gives RAG-grade
vectors (good for retrieval / similarity), not a substitute for a dedicated
embedding model. See dasLLAMA-09 — Embeddings for the facade side.
curl http://127.0.0.1:8080/v1/embeddings -H 'Content-Type: application/json' -d '{
"input": ["the quick brown fox", "a lazy dog"]
}'
# -> {"object":"list","data":[{"object":"embedding","embedding":[...],"index":0}, ...],
# "model":"...","usage":{"prompt_tokens":N,"total_tokens":N}}
7.17.3.3. Transcription (with --asr)
curl http://127.0.0.1:8080/v1/audio/transcriptions \
-F file=@audio.wav -F response_format=verbose_json
7.17.4. Testing
test_openai_server.das (in the tool directory) is a model-gated, JIT-only
conformance test: it starts the server on its own thread, then drives
/v1/models, /v1/embeddings, and /v1/chat/completions over the real
dashv HTTP client. It skips cleanly when the model GGUF is absent; set
DASLLAMA_MODELS_DIR (one of the knobs in the dasLLAMA knob reference) to a directory containing
tinyllama-1.1b-chat-v1.0.Q8_0.gguf, then:
bin/daslang -jit dastest/dastest.das -- --test utils/dasllama-server/test_openai_server.das
7.17.5. Not yet implemented
Tool / function calling (tools, tool_choice) — parked as a follow-up.