GPU Power Limits and Undervolting on Linux
How to power limit, lock clocks, and undervolt Nvidia GPUs on Linux with nvidia-smi, fixing transient power-spike crashes on a 3x 3090 rig.
Henry Freed
Henry Freed's projects and notes: mixture-of-experts research, model pruning and distillation, interpretability, GPUs, and the occasional loaf of bread.
Latest posts
Does the Teacher Matter?
Distilling a tiny NanoGPT-style model from OLMo 3 7B variants to test whether the teacher's post-training, quantization, size, and tokenizer matter.
Variable FlexOlmo
Pruning FlexOlmo's math expert down to ~800M parameters with knowledge distillation to show that smaller experts can still contribute to the combined MoE.
High Performance Whole Wheat Bread
Making 100% whole wheat bread that isn't dense: use fresh flour and treat the bran and germ as a mix-in rather than as flour.
Width, Depth, Latency, and You
Visualizations of how transformer depth vs width affects inference latency on a 3090, and what that means for pruning decisions.
Pruning OLMo 3 7B
Pruning OLMo 3 7B to half its size and retraining with knowledge distillation, following Nvidia's Minitron-style recipe.
Variable Sized MoEs
Testing whether variable-sized experts in Mixture of Experts models beat uniform experts, using a modified version of Megablocks.
Mixture of Experts
Notes on adding Mixture of Experts support to Karpathy's NanoGPT, plus general MoE and MoE interpretability notes.
Open Concept Steering: Building Open-Source SAE Feature Steering for OLMo 2 7B
Training a 65k-feature sparse autoencoder on OLMo 2 7B and building an open-source feature steering demo, following Anthropic's monosemanticity work.