PhD student at University of Texas at Austin 🤘. Doing systems for ML.
I just submitted another paper to SOSP 2025 , and it’s hard to believe it’s been nearly four years since I started my PhD. A lot has changed since my last post about my PhD journey—looking back, I seemed pretty desperate then. So here I am, reflecting on the past few years. I feel far more confident now—not just in my research decisions but in navigating the space of SysML in general. When I…
TLDR; the Stichable Neural Networks paper includes some interesting concepts. It allows the creation of multiple neural networks with varying complexity and performance trade-offs from a family of pretrained models. Key Principles How to choose anchors from well-performed pretrained models in a model family The design of stitching layers The stitching direction and strategy Simple but effective…
This is an archive including blogs I find useful or interesting. Hopefully the updates will keep coming. Hardware Server the home ML Google Research Blog Microsoft Research Blog Ahead of AI AI Snake Oil Huggingface Blog Huggingface Daily Papers Lil’Log : really good ML learning notes. Oxen.ai Language Models & Co. : many great visual illustrations Thinking Machines Blog Security null program…
In the previous post , we’ve explored how to write primitive functions in TensorIR. Here, we will see how to transform TensorIR into other (potentially more performant) variants. The content is drived from the mlc course taught by Tianqi Chen . Batched BMM ReLu A batched matrix multiplication followed by a ReLu operation can be expressed using numpy as: def lnumpy_mm_relu_v2 ( A : np .…
TensorIR is a compiler abstraction for optimizing programs with tensor computation primitives in TVM . Imagine a DNN task as a graph, where each node represents a tensor computation. TensorIR explains how each node/tensor computation primitive in the graph is carried out. This post explains my attempt to implement 2D convolution using TensorIR. It is derived from the Machine Learning Compilation…
Table of Contents Single-Controller Multi-Controller Systems Going Back to Single-Controller Deadlock Solutions to Deadlock Google recently released the paper about its new ML system called Pathways . I’m a bit surprised since I expect it to introduce a brand new model architecture. In fact, this paper is not easy to digest at all. I feel like it’s written for people who spent many…
FlexFlow is a deep learning framework that discovers a fast parallelization strategy for distributed DNN training. It uses SOAP (Sample-Operation-Attribute-Parameter) search space of parallelization strategies. in short, FlexFlow automates the parallelization of model training. The four elements in SOAP search space represent something that can be sliced into smaller chunks. For example, sample…
Recently, I was revisiting materials in Deep Learning. I need tools that generate diagrams easily. Drawing the graphs from scratch and upload them individually to the image hosting platform is a daunting process. This is when Mermaid comes into rescue. Now I can generate diagrams directly using Markdown. Here’s how to do it inside a Hugo site. I use the etch theme, but this process should…
Many deep learning tasks involve classification, where a model outputs a series of probabilities for their corresponding labels. The goal is to correctly predict a given input’s label. Mathematically, it means generating max probabilities for the correct label. The probabilities are generated through a process called softmax . The softmax function outputs a vector \(\hat{y}\), which…
Let’s say we want to classify an input text \(y\) and give it a label \(x\). Formally, we want to find: \[ \textrm{argmax} P(x | y) \] By Bayes’ rule this is the same as \[ \textrm{argmax} \frac{P(y|x)P(y)}{P(x)} \] Suppose we have five documents as training data and one document as the input as testing data. Our objective is to give a label to the test sentence. Credit: Eunsol Choi…
This is my personal list of resources related to machine learning systems. Feel free to drop me an email if you think there’s something worth mentioning. I will try to update this page frequently to include the most recent stuffs in mlsys. Resources Facebook’s external large-scale work NGC Container Doc : great for development, without having to manually install CUDA, pytorch, and…
This is a guide on setting up Megatron-LM with FastMoE . Megatron is a transformer developed by the Applied Deep Learning Research team at NVIDIA. FastMoE enables PyTorch support for the Mixture of Experts (MoE) models. We use the FastMoE layer to replace the MLP layers in the transformer language model. Prerequisites Docker We recommend using one of NGC’s recent PyTorch containers . The…
To install Slurm , we need to have admin access to the machine. This post explains how I got Slurm running in multiple Linux servers. All servers are running on Ubuntu 18.04 LTS. Setup Munge First, we need to make sure the clocks, users and groups (UIDs and GIDs) are synchronized across the cluster. We need to create two users: slurm and munge across all servers. z Then, we install Munge for…
Dynamic Tensor Rematerialization ( DTR ) treats GPU memory as a large cache, where tensors can be evicted to save memory, and recomputed if needed later. DTR’s eviction policy relies on the heuristic \(h\). The heuristic assigns a value \(h(t)\) to each resident tensor \(t\), approximating the cost of evicting the tensor. DTR evicts the tensor with the lowest cost based on the value of…
This paper aims to reduce GPU memory usage during DNN training. Capuchin achieves this goal though swapping and recomputation , using tensor as unit of operation. The major question is how to balance between swapping and recomputation to achieve max resource utilization. Swap and Recomputation Benefit The ultimate goal of swapping and recomputation is to hide the overhead as much as possible to…
Today marks the third month of my PhD life. Things finally start to become a little bit clearer. I finally have some potentially concrete ideas to work on. Finding a research topic was the most difficult part. For several months, I was wondering around like a headless chicken, reading papers after papers: serverless, ML inference, compiler, pathlet routing, RDMA, you name it. The feeling of not…
Update : use ssh key to access the repo is strongly recommended. Recently, GitHub deprecated the use of password for repos. You will have to generate GitHub tokens to access repos. It’s difficult for me to memorize the token without serious efforts. Fortunately, it’s easy to mitigate the problem. After a repo is cloned, simply execute git remote remove origin to remote the old remote.…
In a distributed system, it is common for processes to reach consensus. When all non-faulty processes terminate, we must guarantee that everyone agrees on a specific value. Unfortunately, FLP 1985 1 proved that no asynchronous algorithms could achieve consensus. Why is that an issue? The key lies in the fact that asynchronous communication doesn’t preserve order of message arrivals. To fully…
No systems can provide fault-free guarantees, including distributed systems. However, failures in distributed systems are independent . It means only a subset of processes fail at once. We can exploit this feature and provide some degree of fault tolerance. The problem is, fault tolerance makes everything else much more difficult. The most common fault models is the fail-stop . It means a process…
In a distributed system, eventual consistency provides a weak guarantee that data updates will be reflected in all nodes eventually. However, the downside of eventual consistency is that clients could potentially observe awkward intermediate states. For example, appending numbers to a client may result in states like [10], [10,13], [10,12,13]. Therefore, we need stronger consistency guarantees,…
Normally, having consistent event ordering in a distributed system is hard because we have no common clock. Since we don’t have a common clock to measure with, we rely on logical properties of time in the absence of clock. Here we use causality replation between events. In essence, Causality indicates a clock \(C\) is map from events to time satisfying: \(e\rightarrow e’\) implies…
Mutual exclusion is a common term appearing frequently in computer sciences. In essence, it’s a mechanism of concurrency control allowing exclusive access to some resource (or “critical region”). Token passing is an algorithm for distributed mutual exclusion (DME) and will be our focus in this post. DME specifications usually make the following assumptions: Network delivers…
Before we jump into writing specifications in a distributed setting, we first define what a specification is. I take the definition from the magnificent Ken McMillan : a specification is a statement . A statement describes an abstract view of a program. The view itself is often at an interface, which hides or abstracts internal states. A specification is stated in terms of two elements:…
This paper by UTNS lab appeared in NSDI 2021 . It presents a data-analytics framework that decouples intermediate data from computations. Whiz addresses several challenged posed by current analytics frameworks. The first one is data opacity. Most modern data analytics frameworks relies on MapReduce execution engine. The developer specifies the map and reduce function, which then get submitted to…
This paper by Nadeen appeared in MLSys 2021. It presents an in-network aggregation framework called PANAMA for distributed ML training tasks. PANAMA has two components: (1) an in-network hardware accelerator with support for floating-point gradient aggregation; (2) a domain-specific load-balancing and congestion control protocol. Motivation The primary motivation behind PANAMA is the data-parallel…
Update : The official guide from Hugo is for deploying from public repo. This post is intended for deploying from private repo . This post assumes the user has already setup two separate repositories: a private repository for Hugo source files, and a public repository for GitHub Pages . Note: test Hugo site by executing hugo server in the source code directory to make sure the site is generated…
There are thousands of articles trying to explain what exactly a quantum state is. Many of them boiled down to “the state of a qubit is 0, 1, or 0 and 1 at the same time”. This statement leads to both confusion and misinterpretation. The explanation I found on Quantum computing for the very curious is by far the most elegant and simplest: The state of a qubit is a vector in a…
This post covers the topics mentioned in Writing in the Sciences offered on Coursera . Writing Process The writing process includes three steps: Prewriting Collect and organize information Brainstorm take-home messages Work out ideas away from the computer Develop a road map Writing the first draft Putting ideas together in organized prose Revision Read out loud Cut the clutter Verb check Get…
This post covers how to improve sentence structures, and builds to to writing strong paragraphs. Most contents comes from the Writing in the Sciences course offered on Coursera . Punctuation Here is the list of punctuations ranked based on their power to separate: Comma (,) Colon (:) Dash (-) Parentheses ( () ) Semicolon (;) Period (.) The formality of these punctuations are ranked as: Dash (-)…
This is an overview of the second chapter of Writing in the Sciences offered by Stanford . This chapter focuses on writing with strong, active verbs. Lessons include how to: write in the active voice avoid turning verbs into nouns choose strong verbs get to the main verb of a sentence quickly Active Voice There are three advantages of using active voice: Emphasizes author responsibility Improves…
This is an overview over the first chapter of Writing in the Sciences offered by Stanford . The secret of good writing is to strip every sentence to its cleanest components. Every word that serves no function, every long word that could be a short word, every adverb that carries the same meaning that’s already in the verb, every passive construction that leaves the reader unsure of who is…
Recently, I was trying to get the hang of quantum computing. I found myself in a position where I forgot most of the linear algebra stuff I’ve learned in past semesters. So again, I decide to put them down in hope that some of the knowledge here will stay in my memory a bit longer. General single-qubit Gates Trying to understand unitary matrix in the context of pure linear algebra is, I must…
Border Gateway Protocol (BGP) protocol has a very simple purpose: choose the fastest and the most efficient route to deliver a message from one autonomous system (AS) to another. In layman’s term, BGP is the GPS for the internet. Many contents here are credit to Prof. Mohamed G. Gouda . In a nutshell, BGP informs each router \(R\) how to route packets to an IP prefix \(pf\) (i.e. block of IP…
Since my paper on GPU benchmarking was published, every once in a while, I got emails asking me why Altis doesn’t build on their platforms. It almost always has something to do a small script which is responsible for finding CUDA dependencies. This script is invoked every single time make is executed. For some reason, the regular expression in the script sometimes breaks randomly, depending…
This is a summary over the high-level design of SAT solver covered in Prof. Dillig ’s Automated Logical Reasoning class. It’s meant to cover the basic steps towards determining whether a given boolean formula is satisfiable or not. Convert to NNF The first step in a SAT solver is to convert a given boolean formula to Negation Normal Form (NNF) . A normal form of a formula \(F\) is…
Because of Professor Dillig ’s class , I finally got the chance to try out Dafny , a language made by Microsoft Research , with built-in support for formal specification through preconditions , postconditions , loop invariants and loop variants . I often think, what if we write programs in a verification language, would there be much less bugs and will it make our lives much easier than…
In my previous post , we’ve gone over the high-level structure of blockchain and its attributes. This post covers Ethereum and explore how blockchain can be used not only for money transfer but also application development. More Than Money The idea behind Ethereum was proposed by Vitalik Buterin . He wanted to apply the idea of decentralization to build applications with a central authority…
I’m glad it’s over. I applied for CS Ph.D. programs this past fall and had interviews with schools from late December all the way to March. Now that the semester has ended, I decided to put down some reflections on this process. This post is not intended to be the most comprehensive CS Ph.D. application tutorial in the world, but merely a half-guide half-memoir of journey towards a…
The first time I’ve heard the term “blockchain” was around 2014. Since then, its popularity has grown rapidly. However, I’ve never actually understand what blockchain is exactly, until recently. In fact, I didn’t really understand the difference between blockchain and bitcoin. For me, blockchain is clubbed with cryptocurrencies. So here is a short summary of what…
Hoare logic forms the basis of all deductive verification. To illustrate Hoare logic, we first consider a smaller imperative programming language IMP . In IMP, we have three program constructs: expressions, conditionals, and statements: Expression takes the form \( E := Z\ |\ V\ |\ e_1 + e_2\ |\ e_1 \times e_2 \) Conditional is self-explanatory: \( C := true\ |\ false\ |\ e_1 = e_2\ |\ e_1 \leq…
This is a summary of how to compute congruence closure. I implemented the algorithm to compute congruence closure and thought I’d never forget it. But my memory starts to get blurry just after two days. So I figured I’d put things down so I don’t have to watch the entire lecture again the next time I need it. Equivalence Relation Equivalence relation has three properties:…
This is a summary over program loading, dynamical paging, signal handling, and memory mapping in Linux. execve Syscall One of operating systems’ basic services is to load programs into memory to execute. Programs rely on execve syscall to get the OS to load the program into memory and start it executing as a process. The kernel version we used to testing is 5.4.0. Doing a quick search inside…
This is a summary on scheduler activation. To discuss about scheduler activation, we must first understand what is a thread. A thread of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler. Kernel Level Threads Pros/Cons Good functionality, system wide integration Threads are seen and scheduled only by the kernel. A lot of kernel…
I was using Mathjax v2 for a while and I heard v3 perform significantly faster than v2. Many great tutorials explains explains how to add Mathjax support to Jekyll websites. Some of them only cover Mathjax v2. So here is the brief summary on how to add Mathjax v3 support to your Jekyll website (Recently I’ve migrated to Hugo but adding support to Hugo is also pretty similar). In the…
This is a summary over Linux kernel program measurement and mmap. The specs of our experiment environment is listed below. For more details regarding the CPU spec please refer to cpu world . This is the system spec: Attribute Value Processor name (BIOS) Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz Cores 6 Logical processors 12 TLB/Cache details 64-byte Prefetching Data TLB: 1-GB pages, 4-way set…
VMWare ESX Server is a software layer designed to multiplex hardware resources among virtual machines running unmodified commodity operating systems. ESX Server, different to VMware Workstation , is a type 1 hypervisor, which means it runs directly on bare metal. ESX Server focuses on running guest VMs without modifying the guest OSes at all, which is challenging. Memory Virtualization is done by…
Xen is an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, without sacrificing either performance or functionality. Xen is type I hypervisor, which directly runs on top of bare metal. We will summarize what Xen is what its attributes are. paravirtualization - presents a virtual machine abstraction…
Table of Contents Getting the VM running in KVM Building the Kernel Build and Install Kernel Modules Booting KVM with the new Kernel Booting Process Debugging Kernel Set Breakpoints Syscall This is a summary of how to compile and boot the Linux kernel on the KVM-qemu virtual machine. It covers how to get a VM running in KVM, how to build a customized kernel, and how to use GDB with the Linux…
This research is conducted by Martin Heusse, Franck Rousseau, Cilles Berger-Sabbatel, Andrzej Duda on analyzing the performance of the IEEE 802.11b wireless local area networks. Degraded transmitting rate is caused by CSMA/CA channel access method. Overview The performance of the IEEE 802.11b wireless local area networks have degraded performances when some mobile hosts use a lower bit rate than…
Exokernel is a term every system researcher has heard of at some point in life. However, according to the PDOS group at MIT, there aren’t any exokernel-based operating systems in active use today. It’s interesting to discover what ideas exokernels brought to the OS high-level design and some potential drawbacks of such design choice. Perhaps the most important thing to keep in mind is…