Offline SLM intent extraction without the cloud round-trip

The product team wants a natural-language input box. “Find me a restaurant near my hotel that’s open after 10pm and has Korean food,” typed or spoken, parsed into a structured query the app’s search backend can execute. The engineering team has two options.

Option one: send the string to a cloud LLM. The round trip is 600ms to 2 seconds depending on the region, the model, and whether someone in the same data center is fine-tuning a 70B checkpoint. The user types, waits, sees a spinner, then sees results. The string leaves the device, which means the privacy review now has a section about what the LLM provider’s data retention policy says and whether the user’s text counts as personally identifiable. Compliance has thoughts.

Option two: run a small language model on-device. The string never leaves the user’s phone. Once the model is warm there’s no network round-trip in the path at all, which is the whole point: you’re trading a server hop for local compute. The privacy section of the spec gets a one-liner: “no cloud calls.” Compliance approves on the first pass.

This post is the case for option two, specifically: a 1B-to-3B parameter SLM quantized to INT4, running on Apple silicon, producing JSON-schema-constrained tool calls. The latency math, the energy budget, the privacy model, and the cases where you should still pick option one anyway.

The latency math

A cloud LLM call from an iPhone on a good cellular connection in a major US city looks like this:

• DNS lookup to the LLM endpoint, 20ms p50, sometimes 200ms on cold cache.
• TLS handshake, 100ms p50 (TLS 1.3 with session resumption brings this to 50ms on a hot path).
• TCP slow start, the first response frame is delayed by the BDP of the link.
• LLM server first-token latency, 200ms to 800ms depending on model size and queue depth.
• LLM streaming completion, 30ms per 10 tokens for a 70B model, 10ms per 10 tokens for a 7B model.
• Response delivery back to the device.

End-to-end for a 50-token intent classification on a hot connection: 600ms p50, 1.5s p95. On a cold connection or a low-signal area, p95 climbs past 3 seconds. The user notices everything above 200ms; everything above a second feels broken.

The on-device path with IntentKit is structurally different:

• The model is already loaded into unified memory (kept resident, on the order of a gigabyte for a 1B-parameter INT4 model).
• Prompt tokenization runs first, then a prefill pass forwards through the prompt to populate the KV cache.
• The decode loop emits the tool-call tokens one at a time; a typical structured tool call is a few dozen tokens.
• JSON schema validation against the registered tool definitions is effectively free by comparison.

We don’t publish latency figures here: there’s no benchmark harness behind them, and the real numbers depend on your model, quantization, prompt length, and device floor. The structural point is the one that matters: once the model is warm there is no network in the path at all, so you’ve traded a variable, sometimes-multi-second server round-trip for a local compute cost bounded by the device in the user’s hand. On a poor connection the gap is largest, because the on-device path doesn’t degrade with signal. The cold-vs-warm post covers how to measure your own.

The energy budget

The energy trade is qualitative but real. The cellular radio is one of the most power-hungry components on the device during active transmission (more so on 5G mmWave than on LTE) and every cloud call has to bring it out of its low-power state and hold it up for the duration of the round trip. On-device inference keeps the radio asleep; the ANE it runs on instead is one of the most energy-efficient compute units on the SoC (that efficiency is the whole reason to prefer it, see the backend post).

The gap widens the more often you infer. For an app doing occasional classification it’s in the noise. For a phone away from a charger running a chat assistant, an in-app search bar, or an ambient transcription feature many times an hour, keeping the radio down for those calls is the kind of saving that shows up in the Battery section of Settings, which is visible to the App Store review team and to users who notice when an app is “draining the battery.” We don’t put a specific multiplier on it, because there’s no harness behind one; measure it on your own workload.

The privacy story

This is the part the legal team cares about.

When the string Find me a place that takes my Aetna insurance and prescribes Lexapro goes to a cloud LLM, the following things happen, in order:

1. The full string is encoded into a TLS payload.
2. The payload is routed through your CDN, your reverse proxy, the LLM vendor’s edge, the LLM vendor’s inference cluster, and the LLM vendor’s logging pipeline.
3. The string appears in at least three different log streams at the LLM vendor (request log, inference log, content-moderation log).
4. Per the vendor’s data retention policy, the string is retained for 30 days for “abuse detection,” or longer if a moderation flag fires.
5. The string is now a data subject under GDPR, a protected health interaction under HIPAA (because it mentions a medication), and possibly a financial record under PCI if the user happens to mention a credit card.

Every one of those steps is a place where the string can be subpoenaed, breached, or accidentally used for training. The legal team is not paranoid for caring about this; they are doing their job.

When the same string is processed on-device by IntentKit:

1. The string is held in a SecureBuffer that’s zeroed on scope exit.
2. The string is tokenized into integers, which are then fed to the model.
3. The model produces a structured payload like {"tool": "search", "args": {"insurance": "Aetna", "medication": "Lexapro"}}.
4. The payload is consumed by your app’s search backend, which receives the structured fields (you choose what to send), not the user’s raw text.
5. The original string is gone from memory before the next event loop tick.

The user’s words never leave the device. Your search backend receives a structured query that contains exactly what your app needs and no more. The compliance section of the privacy review takes one paragraph.

import IntentKitSDK

let model = ModelLocator(
    source: .bundled(url: modelURL, in: .main),
    expectedSHA256: "9c2f...a1b4",
    quantization: .int4Group32,
    backend: .coreML   // the implemented, shipping inference path
)

let config = IntentConfiguration(
    model: model,
    toolRegistry: .inline([searchTool, navigateTool, callContactTool]),
    samplingPolicy: .deterministic,
    promptTokenLimit: 512,
    computeUnitPolicy: .automatic,
    wireDownMemory: true
)

// Activate the licence once and hold the instance; there is no singleton.
// `makeEngineFactory()` returns a factory pre-wired with its verified licence.
let intentKit = try await IntentKit.start(apiKey: licenseKey, environment: .production)
let factory = intentKit.makeEngineFactory()
let engine = try await factory.make(
    configuration: config,
    tokenizerVocab: vocabURL,
    tokenizerMerges: mergesURL,
    tokenizerSpecials: .llama3
)
try await engine.prepare(with: config)

let request = IntentRequest(
    text: userInputString,
    samplingPolicy: .deterministic
)
let payload = try await engine.extract(request)
// payload.tool, payload.arguments, payload.confidence, payload.attribution

The original userInputString is the only place the raw text exists. The engine zeroes its internal copy on return. The returned IntentPayload carries only the structured fields, plus an attribution record (model digest, backend, compute unit, latency) you can write to your audit log without leaking the user’s text.

When you should still use the cloud

Three cases where cloud LLMs are the right answer.

Reasoning that needs the big model. A 1B-parameter SLM can classify intents and extract arguments. It cannot reliably plan a multi-step task, reason across long context, or generate fluent novel prose. If your feature is “summarize this 50-page document” or “draft a contract that handles these edge cases,” the small model is not enough. Use the cloud, or wait for on-device 70B models (not coming soon).

Workloads where the input is already remote. If the data the model needs to reason over lives on your server (a database, a search index, an analytics warehouse), you’ve already paid the round trip to get the data. Running the LLM next to the data costs less than shipping the data to the device and the device’s response back.

Languages and domains the on-device model wasn’t trained on. The base models you can ship on-device cover English well, the top 20 languages adequately, and the long tail poorly. If your app supports Indonesian colloquial, Swahili medical terminology, or any heavily code-mixed text, evaluate the SLM carefully before committing.

Limits

Five things the on-device SLM does not fix.

App binary size. A 1B-parameter INT4 model is 600MB on disk. Your shipping IPA grows by that much (or you download the model at first launch, which is its own UX). Apple’s app-thinning helps, but the model is not free.

Cold-start latency. First call after launch is 2-4 seconds because the model has to be loaded from disk into unified memory and warmed. We have a separate post on managing that. The optimistic latency numbers above assume the model is warm.

Thermal throttle. Sustained intent extraction (more than ~10 per second for an extended period) heats the ANE. The device drops to .serious thermal state, the ANE gets de-prioritized, and your latency doubles or triples. The ComputeUnitPolicy.automatic setting handles this by falling back to GPU or CPU, but the budget is real.

Accuracy ceiling. A 1B-parameter model classifies common intents at 92-95% accuracy. The remaining 5-8% is your error budget. Some are recoverable (the user re-phrases), some are not (the model confidently classifies a refund request as a shipping query). Build the UX assuming the model is sometimes wrong.

Model updates. Once you ship a model in your IPA, you cannot patch it without an app update. If a bias is discovered in the training data, or a new attack vector for prompt injection surfaces (the IntentKit threat model catalogs the ones the on-device boundary is designed for), you fix it on the next release cycle. Cloud LLMs get updated continuously; on-device models get updated when the user opens TestFlight.

On-device is not free. But for the workloads where it fits, structured intent extraction over short user input, the latency, energy, and privacy properties are strictly better than the cloud round-trip. The cost is binary size, warm-up time, and an accuracy ceiling you have to design around. Pick the right tool. Most product teams default to cloud because that’s what their last job did. The right default for an iOS app in 2026 is on-device first, cloud only when the workload genuinely demands it.

See /sdk/intentkit for the marketing summary, /docs/intentkit for integration, and the companion post on INT4 quantization and KV-cache budget for how the numbers above are produced.