Recent articles from Dot Net Perls Blog.
I was feeling a little left out over the course of this last week as most of the open-weights models released were too large for me to run locally . Meta started the week with Muse Glimmer, a 30B parameter dense model. Then Qwen 3.8 and GLM 5.3—all of which were a bit too large for my 12 GB Nvidia GPU. Meta Muse Glimmer However, I found a 2-bit quantization of Meta Muse Glimmer , and decided…
I have been experimenting more with various chat bots available on the Internet. Inkling , a new LLM from the American company Thinking Machines, is a higher-end LLM (although not really frontier). I tried having it rewrite a few Rust functions . Inkling Model It reasoned through the statements and carefully designed faster versions. I liked how it gave me several rewritten versions : ones that…
Today I was using an online chat bot (a well-known one) and exploring changes to my Rust code. The chat bot confidently told me that in Rust a Vec of bool is encoded as a single bit per bool . I even asked if this was "truly" the case in Rust, as I was not familiar with this. Rust Vecs In Rust the Vec type does not tend to have type-specific optimizations like for bool . It is supposed to be…
I tested Muse Spark 1.1 , a new close-to-frontier LLM from Meta Superintelligence Labs , on my Rust program. Given that I have used small local models to maintain my Rust program, I felt Muse Spark should have no trouble. I had a fairly simple task I wanted done. Muse Spark I set up the model in OpenCode with OpenRouter . I had to verify that I am indeed at least 18 years of age before it would…
Now that I have an agentic AI system set up (I wrote my own local MCP server) I can test different models on it. However, it became clear in testing that occasionally a model will make a "bad" tool call—it will go on a rampage and start deleting files or inserting data incorrectly. Model Rampage I spent some time working on making safety checks to prevent bad agents from ruining my data. I…
I have mostly concluded my efforts in building (and optimizing) a local MCP server for llama-cpp . Making things fast is really important when designing agentic AI systems. Can you imagine if a chat bot took an hour to return a response? No one would use it—so being fast is important. LFM 2.5 Some optimizations I made: • Used Stdio instead of HTTP to reduce overhead in MCP tool calls.…
An MCP server provides a list of tools , along with their parameters and descriptions, to an AI agent. For example it may have a tool like load_file and a description saying how to use it. This description must be tokenized for the LLM to understand the tool. Minimizing Tokens It follows that using shorter descriptions is going to be faster —there is less text to tokenize and process for the…
I have been trying to develop an agentic AI system with a local MCP server . I am using llama-cpp with the LFM 2.5 2.6B model—this LLM is only 1.7 GB and is very fast. However I found the model used a parameter in one of the tool calls correctly about 0% of the time. Tool Use Other models, including Gemma 4, used the parameter correctly nearly 100% of the time. So I tried to figure out what…
In my Rust program I have code that tries to normalize the arguments passed into it. The idea is that if an AI calls my function, I want the function call to succeed even if it makes a mistake in the syntax. String Copies I had repeated string replace calls, and to normalize the name parameter about 5 passes were done, and 5 copies or more were made of the string. This is not a big performance…
I am making some sort of customer service MCP server for my own use. I can tell an LLM (such as LFM2.5-2.6B-Q4_K_M.gguf in llama-cpp ) what I want, and it will do it for me. I have been thinking for ways for an AI agent to convey to the MCP server what the user is talking about. MCP Hints Basically, if the user says something like "view tutorial XYZ", the AI agent needs to provide a hint about…
After using an LLM in llama-cpp to handle local tasks, I started wondering: why couldn't I write some code to handle the queries faster than the LLM inference? This might work for simple prompts, but it would fail on anything complex. Gemma 4 E4B Natural Language Processing is a field of AI, and Large Language Models are a subset of NLP. The main problem with natural languages is that they have a…
I have been using ComfyUI on my RTX 3060 to generate desktop background images for a while. I have tried different subjects, but my favorite has been cool gradients . ComfyUI Gradients I use Z-Image-Turbo which is based on Qwen and developed by Alibaba (alongside Qwen). I set it to generate images of size 2048 by 1024 pixels, using the dpmpp_3m_sde sampler and the Kerras scheduler. I use 20 steps,…
Writing the descriptions in an MCP server is like writing prompts for the tools. You have to explain what the tool does, when to use it, and what the parameters are and how to use them. MCP Server Bloat Essentially it is like coding in natural language , and it can easily lead to bloated prompts when running an AI agent. I wanted to add support for relative dates, like today and yesterday, so I…
I have been experimenting with optimizing my local-LLM agentic AI workflow. Currently I have Gemma 4 E4B (other models work well too, but this one seems reliable and fast) as the LLM. Reducing Tool Calls In testing my system, I found that a simple query, like "summarize the Kimi-K3 blog post" was slower than necessary because the agent had to call multiple tools to resolve the query. It had to…
I seem to be having more success (and fun) using agentic AI (locally with small models ) than I have had with trying to use larger models for editing code. I think there is a major reason why: I tend to enjoy coding (at least at times), and I tend to dislike managing websites . AI Disliked Tasks It makes sense that using agentic AI—running llama-cpp with a small local model like Gemma 4 E4B…
I developed a local MCP server in Rust that has some useful tool calls for often-needed commands. For example it can pull and compile a program, or load a blog post. It was interesting developing the MCP server, but I wanted to think about why an MCP server is superior to running the command lines directly. Local MCP Server The LLM (and the chat window) is a new interface for computer use . With a…
Large and powerful LLMs are great, but sometimes a smaller , local model is sufficient. For example, with an MCP server, a small local model can call tools and be used to accomplish things. I continued to search for small agentic models and found LFM 2.5 . LFM 2.5 Tools This is a model created by Liquid AI , an American company, and it is focused on agentic tool calling . I downloaded the…
Now that I have written my Rust MCP server, I need a local agent to use it to call tools effectively. I found that various models including Gemma 4, Qwen 3.6 and Bonsai 27B can do this, but I tried a new model release called Nanbeige 4.2 . This is a model from a Chinese company that seems to outperform other models of similar sizes. Nanbeige 3B Nanbeige is a Chinese word that means "bridging north…
I rewrote my MCP server to use Stdio input and output instead of HTTP. An MCP server that uses Stdio still receives and returns JSON-formatted messages, but no HTTP connection is made. Instead, a client (like llama-cpp ) uses a sub-process to interact with the MCP server. Stdio MCP Server In this sense, a Stdio MCP server is not a real "server" but is just a regular console program . In any case,…
Yesterday I developed a simple MCP server , which allows local LLMs to call tools that I have written in Rust code. This worked well, but I ran into CORS errors —CORS is a security feature on web browsers. I found that llama-cpp has a special feature to avoid these issues. Llama-cpp ui-mcp-proxy With ui-mcp-proxy , passed as an argument to llama-server , a proxy server is set up so that the…
LLMs cannot actually do anything except generate text—unless you give them access to tools . In llama-cpp there are some built-in tools that you can enable with the "tools all" argument. However more tools can be used from MCP servers . MCP Servers An MCP server (Model Context Protocol) is an HTTP server that returns a JSON list of tools that a model can call, along with some details about…
Tests are important for ensuring code quality , and are even more important for programming in the age of LLMs . I have been using Laguna XS 2.1 with OpenCode to improve my test cases. Rust has native support for tests with "cargo test", so there are no excuses for skipping unit tests. Writing Tests With AI agents, I have noticed they tend to create unit tests for functions they add. But sometimes…
I am continually interested in having a small, local LLM (running in llama-cpp ) that can edit and add simple features to programs written in Rust and similar languages. Recently support for the Laguna models from Poolside AI (an American AI company) was added to llama-cpp . I tried out the smaller version ( Laguna XS ) locally. Laguna XS I used OpenCode and had the model perform a series of…
Code is basically a burden or liability —it has to be maintained. If a file is moved, a Bash script that depends on that file can fail. The more code you have, the more you must maintain; and this can become overwhelming and turn into a full-time job. AI Markdown I have wondered how local AI agents like those supported by llama-cpp can reduce this problem. This is particularly interesting…
The new model Kimi K3 was released this week, and it is claimed to be near-frontier level quality similar to Fable and GPT. I decided to try it out and see if it could optimize my Rust program . I felt a less specific task, like "optimize this function somehow" was a good test. Kimi K3 I set up Kimi K3 in OpenCode , and made a copy of my program directory so that I could throw away the changes…
It has become trendy to rewrite code using AI agents . In my ongoing quest to become more trendy myself, I decided to try to fix a nagging issue in my Rust code base. I had written the Rust code without doc-comments; for some reason I just used inline comments instead. Rewriting Code I tried to use Gemma 4 12B and Qwen 3.6 35B to rewrite the code directly . With recent changes in llama-cpp , this…
Unfortunately using programs like OpenCode with local LLMs and a mid-range Nvidia GPU can be rather slow. Over the past few days I have been experimenting with llama-cpp and its built-in tools (like edit file, read file). llama-cpp Coding It seems possible to replace a harness like OpenCode with llama-cpp and its built-in tools . This would reduce the large prompts used. Info about the project can…
Recently I have been experimenting with llama-cpp and its Tools , which are provided to the LLM to make system calls. For example there are "read file" or "get date time" tools. llama-cpp Tools I realized that programs like OpenCode are essentially the same thing as llama-cpp with tools. With a prompt, I managed to have Gemma 4 and Qwen 3.6 edit a Rust program file. It is necessary to include in…
Speculative decoding is a huge speedup for local inference of LLMs in llama-cpp (and similar programs). Recently I have tried to adjust the settings for MTP, EAGLE-3 and DFlash to get higher tokens per second on my NVidia 3060 RTX 12 GB GPU. MTP, EAGLE-3, DFlash For larger models like Qwen 3.6 35B and Gemma 4 26B, I have found that using a lower spec-draft-n-max setting (for llama-cpp ) is…
Currently LLMs generate text in an autoregressive way—one token after another. Each successive token depends on the tokens before it. However with DFlash , an innovation created by Z-Lab, we can use diffusion and speed up LLM token generation. DFlash With DFlash, a diffusion model is used to speculate what tokens the underlying LLM will generate next. Then if those tokens are correct, they…
Currently it is not feasible (for most people) to run frontier, state-of-the-art models (LLMs) locally . Data centers are needed to run these large models. However, I remain convinced that local LLMs are necessary and are the future of using AI for software development and other tasks. Local LLMs? If you have a subscription to an AI service that is run from a data center, you give up a lot of…
I spent an hour trying to improve a function in my Rust program . Unfortunately, after all my work, it turned out my changes made it slower, so I had to discard the new version. In dismay, I loaded a local LLM in OpenCode (Qwen 3.6 35B with AutoRound) and asked it for some tips. Agent vs. Human It came up with an idea I had not considered—split the function in two parts and remove a bunch of…
When an LLM is quantized , it becomes possible to fit it onto a consumer GPU. Quantization is an important step in getting an LLM to work on many GPUs. Recently I investigated the AutoRound quantization from Intel. AutoRound In AutoRound, a sophisticated algorithm is used to determine the quantized weights. It claims to be state of the art ( SOTA ) for quantization. In my ongoing quest to improve…
Recently I have been using ComfyUI to generate desktop backgrounds for my Ubuntu system. I have an Nvidia GPU (RTX 3060 12 GB) so I have been able to do it entirely locally. It took me a while to figure out the best ComfyUI settings and models. ComfyUI KSampler In ComfyUI, go to the Templates part of the application. I selected Z-Image-Turbo Text to Image , which is a template of models that can…
There is ongoing hype about how AI will replace manual coding . For example Elon Musk stated about a week ago that "things will move maybe even by the end of this year to where you don't even bother doing coding." AI models will just generate the needed binary directly. Coding Obsolete? After using OpenCode for many sessions and rewriting a Rust program with agents, I am less skeptical about these…
Yesterday I decided I needed a new function in my Rust program. I used 3 different LLMs in OpenCode to try to generate this method. The first model I used (a local model) seemed to have problems and could not generate a working implementation after 1 hour of effort. No Code I then decided just to write it myself. I spent about 30 minutes and had an elegant, reliable implementation at the end (yes…
Recently Cohere AI released an open-weights coding model called North Mini Code . This Mixture-of-Experts coding model has 30 billion parameters and 3 billion active parameters. I downloaded a 4-bit quantization by Unsloth with file name North-Mini-Code-1.0-UD-IQ4_NL.gguf ; it was 15.5 GB. North Mini Code After compiling llama-cpp , I was able to run North Mini Code on llama-server on my first…
Some days I like to make some code changes to make my program faster or more reliable. This also helps me keep my programming skills refreshed —I don't want to forget anything important. Today I reviewed file, and decided to make the changes with OpenCode and Qwen 3.6 35B. Refreshing Skills I started up llama-cpp and OpenCode, and typed my specific prompt into OpenCode. The model made the…
The agentic coding I have done so far has been with OpenCode, but I have been hearing about Pi Code so today I tried it out. It seems somewhat lighter than OpenCode. I thought maybe the Gemma 4 12B QAT model I have (with MTP too) might work better in Pi Code. Pi Code I installed the software on my Ubuntu Linux PC and figured out how to use local models with it—one must add a models.json file…
Until today I have had bad results using Gemma 4 in OpenCode. The 26B model and the 12B model both seemed to perform much worse than Qwen 3.6. However, I tried the QAT version of the 12B model, which is a higher-quality quantization, and used the 4-bit version from Unsloth. Gemma-4 12B I used gemma-4-12B-it-qat-UD-Q4_K_XL.gguf and the MTP file, gemma-4-12B-it-Q4_0-MTP.gguf , which is just 253 MB.…
Recently I made a series of changes to my Rust program with various coding agents , mostly using OpenCode. The LLMs I used were DeepSeek V4 Flash and Qwen 3.6 35B (locally-run). It seems like some unexpected things happened to my code base. Code Familiarity First, I barely recognize it now and don't have the familiarity that I used to have with it. Significant parts were not written by me. And I…
There is a hot function in my Rust program that runs about a million times . Even a small change can cause the entire program to run faster or (more usually) slower. Discarding Code I used DeepSeek V4 Flash to explore many possible optimizations to this function. One of the first optimizations I added helped a bit. Unfortunately, none of the later ones did. It seems even AI cannot bend reality to…
In my Rust program, I implemented some logic to replace a 11 MB file read with a mmap system call. I had changed the file's layout to only need the first few kilobytes to be read all at once, and the remaining parts only when needed. mmap (Rust) I used OpenCode and Qwen 3.6 35B to make the change—the patches were generated entirely locally. After making the change, my program completed about…
Some days ago I added AVIF images to this website. I had been using WEBP images for some time, since they were supported by Safari on Mac. But now AVIF has widespread support, and it is an even more efficient format. AVIF Images Image compression is always a compromise between size reductions and quality . But AVIF, at higher compression levels, tends to have better quality than WEBP. In 2026,…
Recently I have been thinking of ways to encode data in the safest and smallest way. With changing data formats of files, it becomes a good idea to introduce some data redundancy and add error codes to the files. In other words—a tiny checksum that can determine if the data is valid. Checksum This can be helpful on a file format that stores many files together as one. If the computed data…
I decided to try generating some images on the computer in ComfyUI . As noted in previous blog posts, I have an RTX 3060 GPU with 12 GB of VRAM. I was not particularly interested in comic book characters, but decided to try some natural landscapes. ComfyUI Images I used the Python 3 venv module to create a lightweight virtual environment. On the ComfyUI GitHub page, there are instructions for…
I have been thinking again about AI-agent assisted coding (it is a popular topic at the moment). I recently made a few changes to some projects, and though I could have used DeepSeek or another LLM, I decided not to. And it was mostly because I knew what needed to be changed, and it would be faster to just do it myself. Agentic Coding? It seems that using LLMs to "vibe-code" large amounts of code…
C# is a language that has been ceaselessly updated over the past decades. In some ways this is good, as it is not "obsolete," and in other ways it is annoying as one can never "learn" it. Recently the designers of C# posted an article about how they plan to implement unsafe support and make it have memory safety like Rust's. C# Memory Safety I looked at the document, and I found it contained 56…
In 2007 I began Dot Net Perls as a blog . It became popular due to the programming information I had on it; I soon changed the site to be more of a reference source than a blog. Blogging Again However, in 2026, AI-generated text has taken over the Internet, and reference material on programming languages can be generated quickly with little effort. Thus I have decided to move back to being a…
It is not clear to me how "successful" this blog is, and this makes sense because these days, AI-generated text is given first priority. But in any case, I seem to keep posting on this blog, and one pain point for me has been the need to decorate text with images. I try to make it look fancy and interesting. Text on Images However, in interest of keeping the site maintainable, I decided to…