An engineer moonlighting as an academic
I attended the File and Storage Technologies (FAST) conference in Santa Clara last week. Some observations and thoughts: "Art is never finished, only abandoned" The keynote talk had a section emphasizing the importance of benchmarking the various components of the system (e.g., disk, network, CPU, memory, context switching, ...) to help understand the result of application performance. The point…
Log-structured merge (LSM) trees are widely used in key-value stores and as the storage layer of database systems. There is a rich tradeoff space in LSM tree designs which I have been reading recently. Before discussing any particular design tradeoffs (in the coming posts), I am starting with a brief introduction of the LSM tree design. Structure A LSM tree stores key-value pairs. The storage is…
I recently faced some scenarios where I was afraid to put myself (more specifically, my thoughts/ideas) out there because of the fear of appearing stupid (i.e., my ideas not being good enough). I work with some really smart people, so this fear is not unfounded either. I realized that not putting my thoughts out there would hurt more than putting out "wrong" thoughts and pushed through. First off,…
Any large-enough neural network training job (e.g., training a LLM or an image or video generation model) employs parallelism strategies. In fact, such large scale training jobs often employ multiple strategies in conjunction. There are two reasons for using parallelism. First, the models are large enough that they don't fit on a single accelerator's memory and hence need to be split up with each…
Effective training time ratio (ETTR) defines the ratio of time spent in training to the wall clock time. In intuitive terms, it is the goodput of the training job. Any time spent doing non-productive (i.e., non-training) work takes away from the goodput. Non-productive time includes reading and writing checkpoints. It also includes the time that was spent in training but had to be repeated because…
Highs [+] Changed jobs and landed at one I like [+] Learning new and interesting things at work (about carbon emissions and LLM training/inference) [+] Learnt to swim [+] Good amount of travel plus quality time with wife (Big Island, Rocky Mountains National Park, Yosemite National Park, Crater Lake National Park, Mumbai, Goa) [+] Read some good books (Careless People, Character Limit, Hatching…
Unless you have been living under a rock for the last couple of years or so, you already know what a large language models (LLM) is. For the rock-dwellers, here is a quick note: a LLM is a machine learning model (more specifically, a neural network) trained on a large corpus of data (think: all the text on the internet) to predict a sequence of words. The outcome is a seemingly "smart" model that…
I did not realize I was being bullied. Largely because I did not understand the concept of online trolling or bullying. This was 2010 after all. I was living in a boys' hostel with common bathrooms and one of my hostel-mates posted a photo on facebook of me coming out of the shower without a shirt on. It wasn't unusual to come out of the shower without a shirt on. Almost everyone did that. But no…
I have been reading the classic consensus papers (Paxos, FLP) recently and the next one in line is about Byzantine Fault Tolerance (BFT). The goal of this class of algorithms is the same -- have all replicas of a state machine reach consensus (typically about the next input). The difference is in the fault model. BFT algorithms assume the existence of malicious nodes, known as Byzantine failures.…
FLP impossibility result states that there does not exist a fault-tolerant, safe, and live consensus protocol for an asynchronous distributed system. A fault-tolerant protocol is one that can handle failure of some nodes in the system. A safe protocol is one in which all the nodes reach the same consensus value. A live protocol is one that terminates in a finite number of steps. An asynchronous…