Small Language Models are the Future of Agentic AI

I had an interesting call with a founder this week who pointed me to a new paper from Peter Belcak (NVIDIA researcher, ex-Meta/ETH/Oxford). The premise:

“Small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI.”

As someone actively investing in agentic infra and apps, that’s obviously interesting. So I read the paper. I think their reasoning makes a lot of sense and influenced how I think about the future of agentic AI systems.

Here’s the distilled version, minus the academic fluff + some thoughts on implications on the startup world.

Today, most AI agents are powered by very large language models. The LLM does the heavy lifting: deciding when to call tools, breaking down complex tasks, planning, reasoning. Agents just ping a LLM API endpoint in the cloud, which spits back answers.

Those endpoints are generalist LLMs serving millions of use cases. They’re big, slow, expensive, and overkill for many agent tasks.

Belcak & co argue that SLMs (sub-10B parameter models you can run on consumer hardware) are:

• Sufficiently powerful → for the majority of agentic subtasks.

• More suitable → most agents don’t need a generalist brain, they need a reliable specialist for many tasks.

• Cheaper by orders of magnitude → in latency, memory, compute, and $$$.

Most agent workloads aren’t open-ended chats. They’re repetitive, scoped, non-conversational tasks.

That’s where SLMs shine:

SLMs are already sufficiently powerful for use in agents

• Recent advances show that well-designed small language models can meet or exceed the task performance previously attributed only to much larger models

• Benchmarks show that small models like Microsoft’s Phi, NVIDIA’s Nemotron-H, and HuggingFace’s SmolLM2 already match or exceed the reasoning, code generation, and instruction-following capabilities of models 10–30× their size while running dramatically faster and cheaper

• With modern training, prompting, and agentic augmentation techniques, capability (not the parameter count) is the binding constraint

SLMs are more economical in agentic systems

• Serving a 7B SLM is 10–30× cheaper than running a 70–175B LLM, with lower latency, energy use, and infrastructure costs

• Fine-tuning agility: SLMs can be adapted or specialized in hours with lightweight methods like LoRA, making them faster and cheaper to evolve.

• Unlike sparse and wasteful LLMs, SLMs use their parameter space more effectively, making them inherently leaner.

• Systems like ChatRTX show that SLMs can run locally on consumer GPUs, enabling offline, real-time inference with stronger data control and less compute.

Agents expose only very narrow LM functionality

• Generalist LLMs were built to cover a vast palette of skills. But in agentic systems, they’re usually narrowed down through layers of prompt engineering and careful context management, leaving only a small slice of those capabilities in actual use.

Agentic interactions need close behavioral alignment

• For agents to work reliably, every tool call and output has to follow strict formatting rules: matching parameter types, order, and the expectations of the calling code. In these cases, flexibility is a drawback.

• A smaller model trained on one consistent format is not only sufficient, but often more robust and cost-effective than a generalist LLM.

In general, think of it as Lego-like AI: instead of one bloated monolith, you assemble small, specialized models into a modular system.

If SLMs are so compelling, why does the industry keep defaulting to generalist LLMs?

Part of it comes down to momentum: billions have already been invested in centralized LLM inference, and the ecosystem of tools and infrastructure has been built around that paradigm. Training and evaluation are still benchmarked against broad, generalist tasks, not the narrower realities of agentic use. And unlike LLMs, SLMs don’t have the same marketing weight behind them—leaving their advantages in cost, flexibility, and alignment underexposed.

Agentic systems don’t have to stick with one big generalist model forever. Over time, they can transition toward a network of smaller, specialized models—without breaking the agent. The paper outlines a step-by-step playbook for how that shift might look:

1. Collect usage data – Log agent calls (inputs, outputs, tool calls, latency)

2. Curate and filter – Clean the data, strip out sensitive details, and paraphrase domain-specific inputs where needed.

3. Cluster tasks – Use unsupervised clustering to spot recurring agent behaviors like summarization, extraction, or code generation.

4. Select SLMs – Match each cluster to a candidate small model based on capability, licensing, and deployment footprint.

5. Fine-tune specialists – Train models on task-specific data using lightweight methods like LoRA or distillation from larger LLMs.

6. Iterate – Continuously retrain and refine as usage patterns evolve.

The end state is a modular system: many small, cheap, high-accuracy specialists routed together instead of a single expensive monolith.

• How much of the performance gap between SLMs and LLMs is task-specific versus generalizable across domains?

• What verticals are most sensitive to cost, latency, privacy, or offline operation, making SLM adoption more compelling?

• Which agent tasks in your domain are repetitive or narrowly scoped enough to benefit from a small language model?

• Could modular SLMs open new revenue streams such as licensing individual specialist models for industry specific workflows?

• Is there an opportunity to build a developer and data platform for the LLM to SLM transition for companies: collecting and curating usage data, clustering tasks, selecting and fine-tuning specialist models, and iterating continuously—while handling orchestration, privacy, and performance monitoring across multiple small models?

• Is there an emerging market opportunity based on this agentic system design shift? Will this be abstracted away by the model infrastructure providers on an internal routing layer or can agent framework builders capture value by offering more design flexibility?

• Is the ability for a small language model (SLM) to fit onto a consumer device the key premise for enabling on-device agents on mobile phones? There are a bunch of startups in the mobile agent space but they all struggle with cloud inference and privacy issues.