This lab focused on Core AI, Foundation Models, Private Cloud Compute, the Evaluations framework, MLX, local models, Xcode agent workflows, model conversion, availability, fallbacks, and when AI should or should not be part of the user experience.
As usual, the goal is simple: make the questions easier to scan, easier to revisit, and easier to connect with real app development problems.
I tried to preserve the original wording and combine related answers where appropriate. However, some inaccuracies or mismatches are still possible.
Enjoy! And subscribe so you don’t miss the next Lab.
Use evaluations, not guesses.
The panel explained the concrete differences first: the on-device model has a smaller context window and works offline, while the Private Cloud Compute model has a larger context window and supports reasoning. The server model can be a better fit for longer context or more complex reasoning tasks, but it depends on the feature.
The recommended workflow is to run comparative evaluations. Build the same feature against the on-device model and the Private Cloud Compute model, then evaluate both with the same data and metrics. Xcode’s evaluation reports can help compare runs side by side.
Latency should also be measured. On-device avoids network latency, but the panel noted that if network conditions are good, the round trip is not always the dominant cost. Instruments now exposes more Foundation Models metrics, such as round-trip latency, token budgets, and related performance data.
Yes, but the important part is the agent integration layer.
The panel distinguished between chat mode and the agentic / ACP path. If you want the model to do more than chat — for example, manipulate a project through Xcode’s tools — the agentic integration is the relevant path.
MLX can run a local model server, and Xcode can detect locally served MLX models in supported workflows. The panel also mentioned Xcode examples that use MLXLM server and open-code style integrations.
Core AI can run inference for a model, but it does not provide a ready-made server layer in the same way. To use Core AI in an agentic workflow, someone needs to provide the protocol conformance or an OpenAI-compatible language-server layer on top.
Check the model configuration and tokenizer first.
The panel said ending-token issues are common when bringing in new language models. If a model emits tags such as unexpected end markers, the likely areas to inspect are the tokenizer, model configuration, and chat template.
The team recommended posting details to the relevant GitHub repos or developer forums. The MLX language model and Core AI language model packages are open source, so issues can be debugged there with concrete configuration details.
The panel also noted that open-code-style workflows with MLX models do work in practice, so if behavior is strange, it is likely a configuration or template issue rather than a fundamental impossibility.
For the on-device model, use the OS version as the practical indicator.
The panel clarified that the on-device model is part of the OS and updates with OS updates. It is not silently swapped out independently of the OS.
For the server model, the situation is more complicated. The panel’s recommended defense is to treat evaluations as part of the workflow. When a new beta or OS release appears, rerun evaluation suites and compare results against previous runs.
They also advised against overly fragile prompt engineering that depends on exact wording for a specific OS point release. A better prompt should be robust enough to work across model updates where possible.
Yes for the on-device model, but developers need to handle availability and rate limits.
The panel said the on-device model can run in the background and in widgets. However, on iOS the request may be rate-limited depending on system conditions. If the device is busy, hot, low on resources, or doing foreground work, the app may need to try again later.
Foreground user activity takes priority. If the user is playing a resource-intensive game, for example, background LLM work is less important than preserving that foreground experience.
macOS is different: the same background rate limiting does not apply in the same way. Still, quality-of-service and system scheduling should be respected.
The practical advice is to catch the specific error, avoid showing raw failures to users, and retry or degrade gracefully.
Start with prompting and context before jumping to fine-tuning.
The panel suggested first using the app’s own data to build better prompts or in-context examples. Foundation Models can personalize behavior by taking relevant user-approved context as input without necessarily training a custom model.
If that is not enough, developers can bring in custom models, use MLX for training or fine-tuning, or convert and deploy models through Core AI. For image-generation-style use cases, the app may need a custom diffusion model or another specialized model rather than the Foundation Models framework alone.
The Evaluations framework can help with “hill climbing”: make a change to prompts, tools, model choice, or fine-tuning, then measure whether the output improved. Model-judge evaluators can also be used for qualitative outputs such as style or image quality when they are turned into measurable scores.
Foundation Models are useful for language-heavy tasks such as content extraction, content generation, summarization, rewriting, classification, and multimodal reasoning with the new image input support.
They are not always the right tool for every machine-learning problem. If an app needs sub-millisecond decisions, real-time per-frame video analysis, ball tracking, or highly specialized speech / vision / translation behavior, Apple’s domain-specific frameworks may be a better fit.
The panel emphasized that Apple already provides high-level APIs for Vision, Speech, Translation, and other specialized tasks. Those APIs can be more optimized, support more languages or modes, and fit certain use cases better than prompting an LLM.
The practical rule: use Foundation Models when language reasoning is the core of the feature. Use specialized models or frameworks when the task is narrow, real-time, or already covered by a dedicated API.
Start with a small project and learn by building.
The panel recommended Swift Playgrounds and the older “Intro to ML” playground as a beginner-friendly path. That project used Core ML to build an image classifier for a rock-paper-scissors game and taught the basics of datasets and model use.
For Foundation Models specifically, Xcode now makes it easy to open a Playground and start with a LanguageModelSessionand respond call. That gives beginners a fast path to experimenting with prompts and responses.
For deeper learning, the panel recommended starting from concrete examples, using online tutorials and courses, and letting tools or agents help explain unfamiliar concepts. Do not start with the math alone unless that is what motivates you. Build something first, then peel back the layers.
Not everything, but many exportable models can be converted.
If a PyTorch model can be expressed through exportable PyTorch APIs and supported operations, Core AI conversion can be straightforward. When a model uses newer or custom operations, developers may need custom lowering, a custom op, or a custom Metal kernel.
For experimentation on Mac, MLX can be a comfortable path for people familiar with PyTorch. MLX APIs are similar enough that PyTorch users can often feel at home. Many Hugging Face models have already been converted for MLX, and MLX LM supports many models out of the box.
The panel also noted that AI coding agents can help translate model code between frameworks, author Core AI or MLX versions, and write custom kernels as a starting point.
Check availability early, pre-warm when appropriate, and degrade gracefully.
The Foundation Models framework provides an availability API. Use it to detect cases such as unsupported devices, Apple Intelligence being disabled, or other availability reasons.
The on-device model also supports pre-warming. If the app knows that a model request is likely — for example, when a view appears and a model-powered button is visible — it can signal the system to load the model into memory ahead of time.
Fallback UX depends on the feature. Sometimes you can hide the model-powered button. Sometimes you can guide the user to enable Apple Intelligence. Sometimes you can use a server model or a custom model through the LanguageModelprotocol. The key is not to expose raw technical failures as the primary user experience.
Use AI only when it adds clear value.
The panel’s guidance was that AI should not be added just because the technology is exciting. Some tasks are better handled by direct UI, deterministic rules, or specialized APIs.
If AI makes the experience slower, less predictable, harder to understand, or less trustworthy, it may be the wrong choice. Developers should evaluate whether the model meaningfully improves the user’s task.
The panel also emphasized privacy, latency, and user control. If the feature requires sensitive data, network calls, or non-deterministic behavior, make sure the benefit justifies the tradeoff and that the app communicates clearly.
The right use case is one where the model helps the user do something that would otherwise be tedious, difficult, personalized, or language-heavy.
Yes.
The panel highlighted the LanguageModel protocol as one of the most important new extensibility points. If a backend conforms to the protocol, it can participate in Foundation Models APIs even if the model is not Apple’s built-in on-device model or Private Cloud Compute model.
That means developers can connect MLX, Core AI, a third-party provider, or their own server-side model to the same higher-level programming model.
The panel noted that many utilities built on Foundation Models apply to any backend that conforms to the protocol. This makes it worth doing the integration work once, because it lets custom providers participate in the broader ecosystem.
A huge thank-you to everyone who joined and asked practical Machine Learning and AI questions throughout the session. Your questions made the discussion useful for developers working with Core AI, Foundation Models, Private Cloud Compute, Evaluations, MLX, custom models, Xcode agent workflows, availability, fallbacks, model conversion, and local AI.
Question acknowledgments: Fantex, Pichaya, Dro Binin, Eric’s Questions, Ants Crashing, Natasha Prau, and the online WWDC audience who submitted and upvoted the remaining Machine Learning and AI questions.
Finally, a heartfelt thank-you to Shashank, Tao, Marcus, Michael, Louis, Ronan, and the teams behind the scenes for sharing practical guidance on building AI-powered apps across Apple platforms.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.