personal website
Mixture of Experts (MoEs) are very popular. They've been popular for a long time, but I'm currently catching up, so there's a lot of new things here for me. I think we generally all understand why imbalanced experts are bad; both from a learning perspective (experts don't improve if they don't get enough gradients) and a performance perspective (unbalanced routing means many GPUs idle). Recently I…
$$ \newcommand{\on}[1]{\operatorname{#1}} \newcommand{\dmodel}{d_{\on{model}}} \newcommand{\dexpert}{d_{\on{expert}}} $$ MoEs are a new major paradigm for scaling models upwards without necessarily increasing compute costs, and they seem to work quite well. There's a lot of interesting work here, but this blog post focuses on recent work on increasing efficiency of MoE kernels. I'm mostly writng…
Hi, I'm Rachit. I model energy systems (generation, transmission, and loads) using machine learning, and like to write in my spare time. I previously spent time at Stripe , Jane Street , DE Shaw , HRT and in the Harvard NLP group . In college, I trained some LSTMs in Lua torch(7) , and spent some time thinking about statistical inference . Afterwards, I worked as a quant and programmer at DE Shaw.…
Applied research teams slow down not because they lack good ideas but because of simpler issues: unclear metrics, confusing experiment design, poor iteration loops, and vague takeaways. I've been working on applied research 1 in some form for my entire career so far. These are notes for early-career quants, ML researchers, and other applied computer scientists trying to improve their research…
$\newcommand{\on}[1]{\operatorname{#1}}$ I think I should be writing down some of the things I learn a little more frequently, so I'll try with a small note here. I read this paper recently: Training LLMs with MXFP4 . MXFP4 is a 4-bit OCP low precision format most well known for use in OpenAI's gpt-oss models. It uses an E2M1 layout (1 sign, 2 exponent, 1 mantissa bit) for the main data, with an…
Recently at work I had a side quest on the following task: choose a binning scheme for some 1-dimensional data that has an annoying distribution. Quantile binning wasn't great (at the sparse tails it produces very wide bins, and in the center very tiny ones), and equal width was also out. Basically I wanted to be able to specify a number of bins ( n_bins ) and get some "reasonable" bin edges out.…
One of the best things about LLMs is that the cost of writing bash or zsh has gone to 0. Here's a way to get started on a new Mac by setting up uv then using it to drive a bunch of macOS automation: #!/usr/bin/env zsh set -euo pipefail # 0. Install uv if missing if ! command -v uv > /dev/null ; then echo " → Installing uv… " curl -LsSf https://astral.sh/uv/install.sh | sh export PATH = " $HOME…
Gardner was so serious about this learning imperative, so determined that the message would get through, that he wrote the speech out in advance because he wanted “every sentence to hit its target.” What was his message? “We have to face the fact that most men and women out there in the world of work are more stale than they know, more bored than they would care to admit,” he said. “Boredom is the…
macOS has a few UX differences from Windows that can make it difficult to switch from Windows to macOS. Here's some tools and tricks to work around them: Remap keys You can change some default keybinds using the Settings app directly under Customize modifier keys (search for this in Settings). If you're more gung-ho, you can use hidutil (easier with this software tool to generate configs). You can…
When disagreeing with someone about a decision or a fact, it's useful to ask someone why rather than just presenting your own view. It reorients the discussion as the team vs the problem, and gives everyone space to acknowledge why they might be wrong. Curiosity Ted Lasso is back for a 4th season, which made me think of this moment in the first 1 . One of my favorite scenes is when Ted plays darts…
This post is a work in progress. I'll update it as I go, and I might be missing very obvious things (or haven't gotten around to it yet). Feel free to shoot me an email if you want to make a comment. A few months ago I was helping with a Rust-based Llama2 inference project and learned a few things about optimizing CPU SIMD code. One thing I couldn't shake is that codegen in Rust is still pretty…
It turns out it's really easy to add a click-to-expand element (also a dropdown or accordion element) in Markdown + HTML without any JavaScript, since it's part of the spec . This: < details > < summary > Click here to expand </ summary > Surprise! </ details > turns into this: Click here to expand Surprise! Of course you can make this fancier with CSS and whatnot. But the core action is built…
Setting up CUDA on WSL2 probably costs you a little bit of performance but gives you a lot of flexibility - you're essentially running a VM with (some) Microsoft support. Here's the best source: Enable NVIDIA CUDA in WSL . Luckily in the past year or so NVIDIA has started supporting this path, so you're (kind of) in good hands. A lot of advice will be mixed between Ubuntu and WSL, though, so I'm…
Lately I've been tinkering with optimizing Sasha's llama2.rs , a fast Rust port of Karpathy's llama2.c. It takes advantage of Rust nightly's portable_simd , which allows you to emit AVX2 or AVX512 instructions using a relatively clean set of abstractions, and also run inference on a quantized llama2, so it's pretty fast. One of the core loops looks like this: let mask = ( 1 << BITS ) - 1 ; // BITS…
I recently heard that one of my high school mentors, Dr. Peter Reilly, passed away last week at the age of 64. He was a supremely patient, kind, and energizing mentor. I think one of the things that strikes me most is how at the time I didn't understand how rare that is, or how lucky we were to have his help. When I met him, I had been looking for research projects to work on with my friend Vivek,…
In the few areas where I have a clue about what people are thinking about, there's words that make me wonder whether this person is either clueless or assuming I'm clueless. One example is that in New York City, if you ask someone what their favorite type of food is, and they respond with "Asian", it sets off a few alarm bells. I'm not a very sophisticated restaurant-goer, and even I know that a…
Pandas added support for nullable float32 and float64 datatypes in the past few years ( Float32 and Float64 respectively) 1 , but there's a lot of footguns, so it feels kind of bolted on. One example is that while arrays with this dtype can have both np.nan and pd.NA (aka None ) as values, Series.isna() only catches the latter not the former. This is a known bug , but there's no fix yet and it…
When teaching someone how to do something, don't say that "it's easy". If it were easy for them, they wouldn't be asking you for help. Even though you're trying to reassure them that it isn't actually that hard, you're starting from different places. For example, take calculus. Once you've learned some calculus, you have the mental framework, and you know what pitfalls to avoid. Someone…
This one is for the poor soul who is attempting to push their code to Github right now via Jetblue's excellent ViaSat wifi, but confused about why it's hanging: It's because JetBlue blocks connections over port 22 (i.e. SSH). It was previously possibly to add the HTTPS url of your repo as an alt remote, but this no longer works because Github has disabled push on HTTPS auth. Github has a tutorial…
Last updated: November 1st, 2023 Windows Subsystem for Linux (WSL) is my main programming environment, after giving up Windows in 2015. It is surprisingly good. Shrink your virtual HD The virtual hard drive in WSL seems to grow without bound (they grow as needed, but won't release space even if you delete data). After shutting down your WSL instances (i.e. wsl --shutdown ), run this in PowerShell…
Checked in code and commits are highly static artifacts intended for reliability. Engineers are told to make small commits so that they are easy to reason about (e.g., tests should pass before and after the commit). Notebooks, on the other hand, are intended to be inherently flexible. They are a space for thinking and exploring. I probably share 5% of the notebooks I make (and those might need a…
This is probably an idea familiar to most researchers (and really anyone who produces written output for their job), but writing it up as you go has huge advantages over writing it up at the end. This isn't just a time-management tactic, but a thinking tool. I learned this from feedback during my first job, but the short summary is: if you write down your results as you go, you don't yet know…
I completely failed to achieve my goal of writing something non-technical at least once a month, so here we're trying not to fail with abandon . As a sad result of my WSL2 setup dying, I've had to reconfigure my desktop Linux experience from scratch again. As a result I've run into Jeff Dickey's rtx , a software version manager written in Rust. Anyone who manages multiple projects with different…
I spent an unfortunate amount of time writing and re-writing my recent blog post about Jupyter notebooks, mostly double checking that I liked the overall style and tone. After I put it up on the shiny new website, I hesitated and finally submitted it to Hacker News, the message board most likely to care. I put my computer away and focused on brunch with my friends, and a few hours later logged on…
I spend roughly half of my programming time working in Jupyter notebooks 1 . Some is exploring data and building models, and some is experimenting when writing new code. I like notebooks - they make exploration much easier. For data science work, visualization with notebooks is just easier than what came before. However, opinions of notebook-style programming vary wildly, from love to mild dislike…
These are things I always forget but need to be really comfortable working in a remote server. I try to be distro-agnostic when I can. Set up history options in .bashrc : HISTCONTROL = ignoreboth:erasedups HISTSIZE = 100000 HISTFILESIZE = 100000 shopt -s histappend shopt -s checkwinsize PROMPT_COMMAND = " history -a; history -c; history -r; $PROMPT_COMMAND " case " $TERM " in xterm-color |*…
If you're like me and have spun up a t4g.small during AWS's free trial period, you might want to run docker and docker compose . However, documentation on this is pretty sparse. docker-compose vs docker compose Apparently , docker compose is the new hotness and should be used going forward. I'm sure there are many changes under the hood but the headliner is that it's written in Go and…
This post covers the Langevin equation , a stochastic differential equation that models the dynamics of particles in Brownian motion 1 . This covers the ideas used in this reference due to Lennart Sjögren. Langevin Equation In 1907 Einstein published a paper that derived a macroscopic quantity $D$, the diffusion constant, with microscopic quantities : $$D = \frac{k_BT}{6\pi\eta a}$$ where $\eta$…
td { padding: 5px; font-family: monospace; font-size: 1.25rem; } th { text-align: center; padding: 0px 5px; } th.left_column { text-align: right; } figure { margin: 0px 20px; max-width: 50rem; } img[src*="#smaller"] { width: 65%; margin: auto; margin-bottom: 15px; } In class we recently discussed the simplified elastic rod model for polymers, which assumes that polymers can be modeled as an…
This isn't an exhaustive list, and I will inevitably forget some papers. I'll keep updating as a remember, and will probably expand some of the background/contribution sections as I have time, so that they're more accessible. Breaking the Softmax Bottleneck: A High-Rank RNN Language Model [ link ] Background : Language models and NLP tasks almost always use a softmax to compute a distribution over…
Random sampling I've been working lately to implement random samplers from a number of distributions in PyTorch, both on CPU and CUDA. This is a topic near and dear to my heart, since it has caused me a lot of trouble multiple times. Once this PR is merged, I'll post an explanation/notebook of why this is important. Here's a brief summary of the motivation: We want to sample from distributions…
td { padding: 5px; font-family: monospace; font-size: 1.5rem; text-align: right; } th { text-align: right; padding: 0px 5px; font-size: 1.5rem; } th.left_column { text-align: right; } figure { margin: 0px 20px; max-width: 50rem; } img[src*="#smaller"] { width: 50%; margin: auto; } tldr: The ubiquitous isotropic Gaussian prior for generative models doesn't make sense / doesn't work, which motivates…
I'm starting this blog to share research ideas that I have, and some solutions to problems I find along the way. I've been helped immensely by other people's blogs in the past, and want to do the same. Also it'll give me a chance to communicate the way I approach problems, and hopefully people will give me alternative perspectives either by email ( rachitsingh@outlook.com ) or in the comments,…