How to make it both fast and accurate. The recently released ModernBERT model is exciting. It takes several advances from recent decoder-only LLMs (think Llama, ChatGPT) and applies them to the encoder-only model that started it all: BERT. BERT-style models don't generate text but they are adept at understanding it. You can adapt (finetune) them for your custom problems, and they are small which…
Building a memory-friendly KV Cache with static shapes No one wants a slow LLM. Most LLMs run on GPUs and most methods to make them fast are tailored specifically to GPUs. LLMs can also run on Apple Neural Engine (ANE), Apple's efficient ML processor that comes in every new iPhone and Mac. Existing GPU optimizations do not easily translate to the Neural Engine which means you end up leaving speed…
What can we learn from them? Apple's latest OSes include several transformer models that are optimized for the Apple Neural Engine. We'll take a look at how they're implemented and see if there's anything we can apply to our own models. To make that easier, I've cobbled together support for viewing them in Netron—you can try it yourself here . While everyone is talking about AI or GPT, Apple…
CSV + gzip will take you far. So you have some time series data and you want to make it smaller? You may not need an algorithm designed specifically for time series. Generic compressors like gzip work quite well and are much easier to use. Of course this depends on your data, so there’s some code you can use to try it out here . Recently I started working on a way to save Bluetooth scale data in…
Shrinking models for Apple Silicon >New to this, but still curious? Don't worry, I wrote the Primer below just for you. Quantization is often touted as a way to make large language models (LLMs) small enough to run on mobile phones. Despite this, very few of the latest methods are able to use the full power of Apple Silicon on iPhone and Mac. This post introduces a new method of quantization that…