The Case For Using Local LLMs

As with many software engineers, I’ve been experimenting with AI tooling and trying to decide how best to use it (if at all).

On a personal level, I wonder how it’s affecting my thinking and productivity. I also tend to wonder how smart it is to be reliant on these corporations when my own skills and profession are at stake.

Regardless, I’ve come to the conclusion that the tools can be useful. When there’s too much noise and not enough signal I just have to trust my instincts here. And take it from me as someone who has no AI course to sell you and no grifting to do.

Local LLMs seem to provide a good middle ground for interacting with AI. The fact they aren’t as powerful as hosted options might even be a boon. The way I also interact with the for learning and programming endeavours also feels productive.

In this post I’ll talk about my reasons for leaning towards local models, and why you might want to consider checking them out too.

If you want to use hosted models right now, there’s basically no competition: around 3-5 companies are offering AI models served over the internet. I’m not against using them, but I believe for individuals, reducing your reliance on third-party platforms can be useful.

These platforms can:

• Change their prices

• Nerf the models

• Remove access to older models

• Have server maintenance or downtime

For many people and corporations these are non-issues. If your employer is fronting your token bill, then maybe you don’t need local AI.

However for me, being able to run a model on a laptop or at home is important. You don’t need an internet connection, and don’t need to fork out money each month for the privilege of writing software.

Where people normally pushback against local AI are capabilities. You’re paying the big bucks because you’re getting the latest and greatest model, or getting all of the newest features (tool calling, web search, etc.).

For my use cases, local models seem to satisfy my needs.

I was actually quite surprised at how good local models are, and for me, they don’t really feel all that different from the hosted options out there.

For reference, I’m using a MacBook Air M4 with 16GB RAM. Somewhat powerful, but nowhere near the beefiest machine in existence. I’m having no problem running local models and getting coherent responses from them.

There is maybe a reason for my lack of issues using local AI, and I want to be honest about that: I do not use LLMs as a coding agent.

Instead, I like to use them as a rubber duck to ask questions and explore decisions around architecture, to help me think about problems, learn new concepts, and occasionally generate code snippets.

People say code is “just syntax”, but it’s also semantics too. I believe writing the code by hand and understanding why things work/don’t work is important, and I try to think about problems as much as I can before relying on AI to answer.

This is not unique thing to AI. People have been copying and pasting code from StackOverflow for probably decades. Your mindset towards learning and understanding the systems is the differentiator.

For me, this means using local LLMs tend to be more than enough for me. I don’t want my skills to atrophy and I’m driven by curiosity. Maybe I’ll eventually come around.

My workflow is a pretty simple one. I use Ollama to install and manage local models, and create my own version of a model with custom settings using a Modelfile.

For example, you can download one of the small gemma4 models with:

Then create a file called Modelfile, and set up something like this:

For me, setting the context at 32kB seems to work well. Beforehand, running gemma4 seemed to create massive lag spikes and stuttering. Since setting the context, I haven’t had any issues.

To creat the custom model, you can run:

I also disable thinking mode as it takes a lot longer to generate a response. You can’t seem to do this directly in the Modelfile yet, so when I run the model I do:

As for actually working, I like to mostly use the terminal with Neovim open in one split, a chat session in another, and then a small window to run terminal commands:

You can also just keep everything in separate terminal tabs if you prefer it that way or want to save on the screen space.

It’s not super complex, and I try to use AI only when I’ve exhausted my other options.

I also use Zed as well if I want a more rich experience, and tend to switch between the two. It’s an excellent editor. You can also set up Ollama as the LLM provider for your AI assistant in Zed, too.

If you haven’t tried it yet, I recommend giving it a go. It’s quite a fun way to work, and I don’t feel the despair when building software that I know some engineers are feeling.

Otherwise, I’m sure there’s ways I could improve my workflow with local AI, whilst also maintaining integrity.

I’m open to any recommendations or suggestions, so feel free to reply if you’re using local LLMs in an interesting way.

Thanks for reading!

None of my posts are paywalled, and are completely free to access. Subscribe to my newsletter if you want to read more articles on FOSS, programming, and self-learning.