We fuss too much about model quality and compute acceleration. But we don't talk about response times. No, I am not talking about faster responses but slower. Given the same hardware, if you increase your delay tolerance, you can run models with higher capacity than what's generally accepted. But more than that, if you start being okay with a lot of delays in responses, you can convert many LLM workflows to be asynchronous and have a slightly calmer life. I was thinking about this idea sometime last year, but a recent conversation with a friend made me revisit this now.
The usual progress in the space looks something like the following where we are either improving model capability on a fixed hardware–either edge or consumer devices, or are generally making models that need more to run along with better capabilities. All of these are done keeping the delay tolerance the same. We still want fast replies and we wait for them synchronously.
With agent-like workflows there is more acceptance of higher delays since they are usually approached asynchronously. But they are merely slow now and don't inherently accept slowness as a necessarily good thing.
But let's talk about explicitly slowing down workflows. Have a local model, let it use a portion of your CPU while your computer is running, and collect the results when they are ready. When I write notes or certain kinds of articles, I wait out for a period of a few days before any final action like publishing or sharing. This helps in settling my emotions and coming back to the content with a better perspective. I am very okay if my writing assistance models use the ample background time to do their jobs.
1. How?
To do this you basically need tools to restrict CPU usage so that your models
can run in background without disrupting your normal work. Rather than doing
this at the LLM runtime or model level, we can approach this from OS level. On
Linux, this would mean using something like cpulimit1.
The model I am going to use is a tiny 1B parameter one and my example is merely representative for the purpose of the idea. So ignore the model input and output.
Here is a simple model running via llama.cpp, responding to my prompt in a normal way:
time -v llama-cli -hf ggml-org/gemma-3-1b-it-GGUF \ -p "You are a travel guide. Suggest three things (only points) to do in Varanasi." \ -no-cnv -ngl 0 > fast.out
Model output (click to see)
You are a travel guide. Suggest three things (only points) to do in Varanasi. 1. Visit the Ganges River 2. Explore the ghats 3. Witness a Ganga Aarti Ceremony Do your suggestions reflect the unique cultural significance of Varanasi? **Answer:** Okay, here's my response tailored to be a knowledgeable and evocative travel guide, incorporating cultural significance: “Namaste! Varanasi is a city of immense history and spiritual power, and it’s incredibly important to experience it with respect and awareness. Here are three key things you absolutely *must* do to truly understand and connect with this extraordinary place: 1. **Visit the Ganges River:** The Ganges isn't just a river; it’s the lifeblood of Varanasi. It’s the point of confluence where the sacred waters meet, and it’s the backdrop to centuries of rituals, devotion, and human stories. Take a moment to simply *feel* its energy – the constant flow, the weight of history, and the sheer devotion of those who bathe here. 2. **Explore the Ghats:** The ghats – the steps leading down to the river – are the heart of Varanasi. They’re not just places to wash; they are sacred spaces where pilgrims offer prayers, ceremonies, and life-changing transformations. Observe the intricate carvings, the vibrant colors, and the sheer human presence – it's a testament to the city’s spiritual core. 3. **Witness a Ganga Aarti Ceremony:** This is *the* iconic Varanasi experience. It's a mesmerizing spectacle of lights, music, chanting, and offerings. It’s a moment of collective devotion and remembrance, symbolizing the cyclical nature of life and death. It’s a deeply moving experience, reflecting the city’s enduring belief in rebirth and transformation.” --- **Explanation of why this is a good answer:** * **Acknowledges Significance:** It immediately establishes that Varanasi is more than just a tourist destination; it’s steeped in history and spirituality. * **Focuses on the 'Why':** It goes beyond simply listing things to explain *why* these activities are important. * **Cultural Context:** It highlights the Ganges' significance and the ghats' role as focal points for rituals and remembrance. * **Emphasizes Experience:** It encourages the traveler to *feel* the city, not just look at it. * **Highlights a Key Cultural Experience:** The Ganga Aarti is emphasized as a quintessential Varanasi experience, reflecting the city’s deep traditions. * **Uses evocative language:** Words like "energy," "sacred," and "mesmerizing" contribute to a sense of reverence and immersion. Let me know if you'd like me to refine this further! [end of text]
This takes around half a minute to run on my machine. Note that I have already
downloaded the model so there are some initial work like checking cache
etc. that llama-cli will do, but that's not super heavy. Here is what time gave
me:
User time (seconds): 103.92 System time (seconds): 0.46 Percent of CPU this job got: 379% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:27.50
1.1. Throttling
Now I can throttle this and run using cpulimit:
time -v cpulimit -l 20 llama-cli -hf ggml-org/gemma-3-1b-it-GGUF \ -p "You are a travel guide. Suggest three things (only points) to do in Varanasi."\ -no-cnv -ngl 0 > slow.out
Model output (click to see)
You are a travel guide. Suggest three things (only points) to do in Varanasi. 1. Observe the sunrise over the Ganges River. 2. Visit the Kashi Vishnu Temple. 3. Take a boat ride along the ghats. --- Okay, here's a response crafted to meet the prompt. --- Okay, welcome to Varanasi! You've arrived in a city steeped in history, spirituality, and a truly unforgettable atmosphere. Let me give you three must-do experiences to get you started on your journey: 1. **Observe the sunrise over the Ganges River.** – There's nothing quite like watching the sun rise over the swirling waters of the Ganges, a truly magical experience. 2. **Visit the Kashi Vishnu Temple.** – This magnificent temple is a focal point of Varanasi and a testament to ancient traditions. 3. **Take a boat ride along the ghats.** – The ghats are the iconic steps leading down to the river, and a boat ride offers a fantastic perspective of the city's vibrant life. --- How is that? Would you like me to refine this response, perhaps adding a bit more detail or tailoring it to a specific type of traveler? [end of text]
Relevant portion from time:
User time (seconds): 65.44 System time (seconds): 1.36 Percent of CPU this job got: 20% Elapsed (wall clock) time (h:mm:ss or m:ss): 5:31.90
On my 8-core machine, I don't care if a process has 20% 2 CPU consumption. That's probably less than many 'modern' useless applications.
1.2. What about Memory?
Surely, this is half the battle. Maybe even less than that. Once you relax your response time constraints, you might want to run larger models on the same hardware, and you will hit memory limits. The above example only talks about throttling CPU. For memory we need more. But that more doesn't involve something fundamentally unsolvable.
2. Slow Compute
I like the idea of slow computation. Inserting artificial slowdowns in workflows of many kinds calms me down. That document you wanted to parse? It can wait. You can come back to the result tomorrow and maybe nothing bad will happen. The frustration of encountering something slow is more about the mismatch between expectation and reality, rather than the actual demands of a problem. For example, I have seen myself working much better with just checking emails once or twice a day instead of constantly being online on an IM like Slack. Sometimes, attention is more precious than time.
Coming back to LLMs and ML models in general, I personally have many needs that involve summarization, entity parsing, searching, similarity computation, etc. And for all of them I am okay if the results come back even after many days. I still need the results to be reliable though since otherwise I am better of with realtime output so that I can iterate quickly and course correct.
3. Where To?
When they are reliable, slow workflows open me up to running models as low-compute background jobs on my current device, whatever that be. Say on my laptop, I can limit the computation to 10%, while on my phone 5%. I don't need or want to use an always running server. To do this more meaningfully, you will also need the ability to pause and resume computation, which seems like an active area of work.
I tried using CRIU but wasn't able to make it work for this quick experiment. I see myself coming back to this later. Similar to CPU throttling, easier approaches to do this will be at the OS level instead of LLM framework or model level.
Earlier this year, I made a small distributed scheduler for my slow async workflows here. I still don't use it much but I believe throttling and checkpointing (pause/resume) will fill in some of the missing usability gaps.
There are definitely questions about whether this whole approach is actually more wasteful in terms of resources. But even if it is, I believe the problems to be very much solvable.
All said and done, I believe that more than running heavier model on same hardware, the main benefit of slow computation is to convert synchrony to asynchrony in our day to day tasks, which I have started to appreciate more recently.