Do sweat the small stuff

Almost two years ago, I wrote this post about small LLMs that you can install on your own machine. Like everything in the world of LLMs, the small end of the market has seen plenty of developments since then, so it’s time for a quick catch-up.

The models I highlighted back then are still alive and kicking, all numerous releases further along in their development cycles. This includes Meta’s Llama, Google DeepMind’s Gemma and Alibaba’s Qwen. These remain very popular amongst the install-your-own-LLM crowd, with Qwen having particularly impressive download stats at HuggingFace, which remains the main repository for open models. Some notable additions since then are OpenAI’s GPT-OSS and DeepSeek’s offerings.

Most of these model families now offer multimodal models for local deployment, meaning that the same model can be used for both text and image comprehension tasks. So you can now insert an image, photo, or a mixed-media PDF into these models, and ask questions about what they see in it. However, there can be a trade-off here, since adding image support may degrade performance in other areas1.

Partly in response to this trade-off, it’s increasingly common to find specialist models. For example, dedicated vision models (e.g. Llama Vision, Qwen-VL) can now compete against much larger generalist models at tasks like image recognition, OCR and captioning. Likewise, specialist coding models (e.g. DeepSeek-Coder-Lite and Qwen-Coder) offer the code comprehension abilities of large LLMs in small packages. This supports a workflow where small specialist models can be switched in and out to solve a variety of tasks.

Small models have also become more and more competent. This means they can be used productively on devices with increasingly modest resources. The introduction of Google’s AI Edge Gallery (available for both Android and iOS) was a milestone here, allowing users to download and run open LLMs within the comfort of their own phones. This mirrors the efforts of tech companies such as Apple to develop models that are small enough to be deployed on mobile devices.

In my previous post, I talked about LM Studio. This is still a popular means of deploying open LLMs, and now includes RAG support, allowing locally-hosted LLMs to interact with locally-stored information, and the LM Link app, which allows locally-hosted LLMs to be accessed from a nearby phone. For those who prefer open source, Jan and Open WebUI offer not-too-dissimilar experiences.

Two years ago, the only frontier model that included the things needed to train it from scratch, including training data, was the Allen Institute’s OLMo. This is still going strong, with version 3 released November of last year. Added to this is LLM360, a community-based initiative that has released various models, including their training snapshots, for those who want to know more about the learning process. At the smaller end of things are the Pythia suite from not-for-profit EleutherAI and HuggingFace’s SmolLM, both aimed at research.

And it’s not just consumers, developers and researchers who can benefit from small LLMs. They’re also a boon for preppers, who are building solar-powered, offline “AI survival kits” to run compact LLMs in the event of civilisational collapse. After all, who needs a wind-up radio when you can query a local LLM plumbed into an offline copy of Wikipedia for survival advice?