Experimenting with local GGUF LLMs & llama.cpp server

I spent some time this weekend experimenting with smaller LLMs locally to get a sense of how they compare to the models I use at work (Claude, GPT, Gemini, etc.) and how they perform on a GPU-only machine. I started with Qwen3-4B . Turns out, running a model locally with llama.cpp server is actually very simple with the GGML project. 1 Here&rsquo;s my compose.yml file: 1 version : '3.9' 2 3…