Recent content on Nitin Gupta
Summary 
 
 Problem : Multi-model workloads are the norm: A/B tests, customer fine-tunes, safety variants, multi-stage pipelines. GPU memory scales linearly with model count, and VRAM is the limiting resource. 
 Solution : Tensor deduplication automatically identifies and shares bit-identical weight tensors across models, requiring no checkpoint modifications. 
 Results : Across…
I keep running into the same pattern when trying to self-host models (which is a lot of fun): we run several big models side by side, all of them valuable, all of them slightly different, and all of them wasting VRAM by reloading nearly the same weights. 
 This post is my attempt to explore a specific idea: 
 Can we load a shared backbone of weights once on a GPU, then load only the small,…
This feature has now been accepted and merged in the upstream kernel and will be part of kernel release 5.9. This post has been updated to match the upstream version of this feature. 
 
 In my previous post , I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel,…
Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency. 
 To show the effect of on-demand compaction on hugepage allocation…
We built an object store from scratch in Elixir using a layered design approach. The overall theme has been to avoid generalizing the design too much which kept implementation of each layer/module simple. We were also careful when adding any third-party dependencies which has multiple advantages: deeper understanding of your codebase, easier debugging (I hate unknown code-paths in backtraces).…
Part I , introduces the overall design of our object store. In this post we focus on the Web layer. This is the final layer for our object store responsible for exposing it over the web. It will expose endpoints: /upload for uploading a file and /file/:file_id for getting a file by ID. A typical GraphQL application with also expose endpoint /graphql which directly plugs into your API layer,…
Part I , introduces the overall design of our object store. In this post we focus on the API layer. All layers till now were just concerned about storing the input file together with some file-format specific transforms (like thumbnails). It is at the API layer where we will be storing per-file system and user metadata. This metadata can be used to support application specific business logic and…
Part I layer. 
 ImageStore 
 The ImageStore module is responsible for storing images along with their thumbnail. It will use the FileStore layer to actually store files on disk. Before we define module interfaces, lets see our application requirements: 
 
 All images must be stored in the jpg format. 
 Images cannot be larger than 1920x1080. We do not want to store user…
Part I , introduces the overall design of our object store. In this post we focus on its first layer, the FileStore . 
 The FileStore layer is responsible for actually storing the file in our object store. At this level, we are not concerned about what kind of file it is (image, video, document, or whatever else), nor do we have any notion of security. We just store whatever input path is…
I recently designed an object store from scratch in Elixir. It has been serving me well as a backend for an app which needs to store all kinds of files: images, videos, documents. I wanted something simple to avoid dealing with off-the-shelf object stores which require complex configurations and to avoid cloud storage which is dead simple to use but can get very expensive, very quickly. For this…
Elixir is a function programming language that I have been using a lot in recent months to build all kinds of applications. Understanding of built-in collection types is essential to use any language effectively and Elixir is no different. 
 This posts summarizes all collection type along with pros/cons/gotchas for each one of them. 
 
 
 
 Collection 
 Example 
 When…
I recently stumbled upon a subtle bug in a benchmark code which again reminds me to never use C++ again, if I can. 
 Here’s a buggy snippet from this code (simplified): 
 // BUGGY
 ostringstream os;
 int i = 1 ;
 os << 'foo-' << i << '.dat' ;
 const char * filename = os.str().c_str();
 int fd = open(filename, O_RDONLY);
 You may expect above code to try open a…
For some time I’ve been looking for a backup solution for Linux that can periodically take snapshots of data, allowing me to go back in history of any file just like git. I finally found restic which fits these requirements. Here is how I set it up to take snapshots of particular directories, say every 15 minutes. 
 Installing restic 
 Though restic is available in repositories of…
There is no official Google drive client for Linux. I tried many different clients found all over GitHub but none of them worked reliably for me except rclone . I also tried third-party proprietary clients like Insync but allowing read-write access to all your Google drive files to a closed source blob is too much to swallow. 
 Once caveat with rclone is that it does not natively support…
Often, you have more than one system at your disposal but no clear way
of distributing your compilation workloads over to all or some of them.
They might be running different OSes which makes it look even more
difficult. In my case, I have one laptop (2 cores) and a desktop (4
cores) connected with a WiFi network. The laptop runs Linux (Fedora 13
64-bit) while the desktop runs…
Recently, I developed Linux kernel driver which creates generic RAM
based compressed block devices (called zram ). Being RAM disks, they
do not provide persistent storage but there are many use cases where
persistence is not required: /tmp, various caches under /var, swap disks
etc. These cases can benefit greatly from high speed RAM disks along
with savings which compression…
Since a long time, I was looking for a graphical git diff viewer which
could show original and modified file side-by-side and highlight the
changes. There are few solutions but none of them is sufficient: 
 
 A tool included with git called ‘git-difftool’ is partially helpful
– it can show changes graphically but diff for each file is shown
one-by-one.…
You worked on some part of Linux kernel. It works great. Now, how to
generate the patch series and send it out for review? For this, I always
used to generate diffs, create a set of draft mails (one for each patch)
in KMail or Thunderbird, and send all these mails one-by-one. This
workflow quickly became a big headache. Then I learned Git (and some
related tools) to do all this…
In case you are unfamiliar with ccache, its a “compiler cache”.
Compiling is primarily CPU intensive task. So, ccache caches compiled
objects - so next time we compile same code, it reuses these objects
thereby significantly 
speeding-up compilation. 
 I need to recompile Linux kernel usually several times a day, with
different permutations of config settings.…
Linux kernel has few SLAB allocator variants included: SLAB, SLUB and
SLOB. Of these, SLOB is especially meant to be used on embedded devices
– it tries to be more memory space efficient than other SLAB
variants. 
 Yesterday, I had a detailed look at SLOB allocator for possible use in
 compcache poject and found it
unacceptable for the purpose. I did it in response to…
Very old but still as relevant… and very interesting too! Directly go
to “anti-tip” section of this article . 
 “The moral of the story is: don’t get tricky. C programmers often try to minimize the number of lines of C in their program without consideration for what the compiler will generate. When in doubt, write clear code and give the optimizer a chance to…
One of my Fedora 10 systems used to freeze very frequently. After lot of
looking around I found its because of “KWin Composing” which gives
OpenGL driven special effects for desktop. Unfortunately, Linux has
always been bad at radeon drivers, so it better to disable these effects
especially if you have radeon video cards. 
 in ~/.kde/share/config/kwinrc: 
 in…
Here is link to paper
( pdf )
( MP3 ) 
 Recently I came across this paper published in OSDI
‘08 . Its an extension to VMware’s
page-sharing and shows some amazing and hard to believe results. VMware
page-sharing mechanism scans memory for all VMs and maps pages with
 same contents to a single page.
This achieves memory savings if multiple VMs are hosted…
I am deeply passionate about optimizing GPU performance and delving into the intricacies of resolving bottlenecks within render and compute workloads. 
 With nearly 15 years of experience delving into the nitty-gritty details of technology, I’ve dedicated a significant portion of my career to working on various low-level components, including Linux Kernel Proactive Compaction ( LWN.net…