Aman Gupta's Blog
I just came back from a vacation to Japan with my wife and 10-month old. We went to Kyoto via Osaka and spent 6 days in Kyoto. It was a lovely experience, although just after we left there were floods in Kyoto due to a double(!) Typhoon hitting Japan. Everything seems to be okay now though. Anyway, what I wanted to say: the most special thing about Japan are it's people. They are kind and…
The MTP PR got merged into llama.cpp a few days back. When I wrote it I did not expect it blow up like it did. It is now the most "liked" PR in the llama.cpp repo. What bigger compliment can you expect when Georgi publicly compliments your work : I agree with Georgi's sentiment- Qwen3.6-27B is truly a pivotal moment for local AI, it's a model that "just works". The Qwen team really cooked with…
This post about about a PR to llama.cpp which adds support to prefetch weights, overlapping compute for the current layer with the weight loading for the next layer. This is useful for memory-constrained environments where the model weights don't fit into VRAM. This is the basic setup: Timeline ---------------------------------------> [Compute Layer 1] [Compute Layer 2] .... [Prefetch Weights-1]…
Autoresearch is all the rage, it's like people like never heard of while loop before. Here it is, not even simplified: let mut baseline = measure (); let mut experiment_log = log :: new (); loop { llm . make_changes (); let new_metric = measure (); if new_metric . is_better_than ( baseline ) { baseline = new_metric ; } experiment_log . record_experience (); } You don't even need to code this up as…
I recently needed to delete objects in a LIFO manner and employed an LLM to rubber-duck about what would be the best container. All of them said std::vector . This confused me so I posed a simpler question to all leading LLMs, and it seems like they all think std::vector destructs elements from back to front. Firstly, we can demonstrably prove this is false using godbolt , here's the simple…
Note: this is a copy of the discussion here LLM inference is divided into 2 phases: prompt-processing (PP) and token-generation (TG). Prompt processing is when entire prompt is fed through the model, whereas token-generation is when the model starts outputting one token at a time. These are different workloads, for decoder-only auto-regressive LLMs PP is compute-bound and TG is memory-bound. This…
There are a lot of resources online about writing a fast GEMM, and they all get complicated really fast, and by the time you reach the tensor core section you need to keep a lot in context to understand. This post tries to go in reverse, using a tensor core on the smallest possible tile and build up from there. It's aimed at software developers who are interested in learning about tensor cores but…
I have a git repo where I put in my life goals. Every time I achieve something, I make a little edit and push a PR. It's just one markdown file, divided into various sections - personal, professional and fitness. As I read through them today: Some of the far-reaching goals I achieved now look way less challenging in hindsight Things I didn't do, I will probably never do I can aim higher Personal…
Contributing to open-source projects is a goal of many programmers. Issues tagged with "good-first-issue" is one way to find something to work on. When newcomers (people who never contributed to the project) browse issues in a repo, GitHub will hit them with a banner "If you're ready to tackle some open issues, we've collected some good first issues for you.", which will take you to an issues page…