Parameter-efficient fine-tuning in tinygrad Home Parameter-efficient fine-tuning in tinygrad Parameter-efficient fine-tuning (PEFT) is a family of techniques used to adapt LLMs (along with other types of large models) to specific tasks or datasets. For example, if you had a private dataset from your legal practice, you could take gpt-oss off the shelf and customize it to be really good at your…
Nix dependency tree Home Nix dependency tree This tip is so useful to me but simultaneously so hard to find that I’m going to dedicate an entire entry to it. How to find all the dependencies a flake output pulls in: $ nix path-info .#bpftrace-llvm18 [..] this derivation will be built: /nix/store/ada4fcj69rqqqxvdzg3f0gvwm6pmzrch-bpftrace.drv $ nix-store --query --tree…
Flaky tests, or: why not to ignore mysteries Home Flaky tests, or: why not to ignore mysteries I spent a few weeks earlier this year tracking down a set of flaky end-to-end tests where bpftrace would occasionally cease to print output. I had gotten as far as figuring out std::cout had badbit set after a write but had run out of ideas on how to debug it. At the time, because I could not reproduce…
Application blackbox Home Application blackbox In aviation, a “black box” is the colloquial term for the flight recorder , a device that records the recent history of a flight. This device is designed to be robust to a potential crash such that the recovered data can be later used during investigation. blackbox applies this idea to software. The principle goal of blackbox is to enable applications…
Big strings in bpftrace Home Big strings in bpftrace If you’ve ever written BPF code you’ve probably run into the limits of the BPF programming model. bpftrace, a high level tracing language built on BPF, does its best to hide these limits – usually through language design but sometimes through clever code generation. Even so, some fundamental issues still leak through. One such example is with…
Reading local variables with bpftrace Home Reading local variables with bpftrace Most tracers are designed around functions. You attach a probe to a function and then when the function is called and the probe is run, you access the function arguments and/or the return value. Function arguments are sufficient for the majority of use cases, as function bodies typically process and manipulate the…
errno and libc Home errno and libc A few weeks ago I wanted to confirm if errno was a libc abstraction or a kernel feature. The glibc docs are deliberately vague on the topic, so experimentation seemed like the best course. Using libc static int use_wrapper ( int cmd , union bpf_attr * attr , unsigned int size ) { long ret ; /* Clear errno */ errno = 0 ; ret = syscall ( __NR_bpf , cmd , attr ,…
Innovative –help messages Home Innovative –help messages Last week I heard about how Google Gemini was offering a very generous free tier on their API. So yesterday I wrote a small CLI application to interface with Gemini. It took about 30 minutes to get something reasonable working. I was satisfied - but something was missing. Fortunately, I woke up with an idea to innovate on a crusty old staple…
Pull request review: still files! Home Pull request review: still files! It’s been about a year since the core prr features features went in. From my perspective, one surprising thing is that there’s now a small group of dedicated users who not only report bugs, but are also willing to implement new features! Speaking for myself, the only thing better to a maintainer than happy users are happy…
Kernel IPIs Home Kernel IPIs An inter-processor interrupt (IPI) is a way for one processor to signal to another processor that some work needs to be done. I’m not sure why I got curious about this today. But since I dug into it, I thought I’d write it down. Interface Since it looks like there’s quite a few IPI vectors, I narrowed down the scope by only looking at a fairly general user:…
Packet drop reasons Home Packet drop reasons Figuring out why the kernel is dropping packets can be challenging. Fortunately, newer kernel releases contain dropreason infrastructure that can make the problem more tractable. We can easily tap into dropreason by using bpftrace. Consider this one-liner: $ sudo bpftrace -e 'tracepoint:skb:kfree_skb { printf("%-18s %d\n", comm, args->reason); }'…
Truthiness in C Home Truthiness in C This week someone mentioned that C supports casting from int to bool . That naturally triggered my curiosity – what does the generated code look like? First I think it’s important to point out many casts in C are “free”. “Free” in the sense that the compiler changes its internal understanding of an expression. But in actual assembly nothing really changes –…
AppImage explosions Home AppImage explosions In a previous post , I detailed how bpftrace’s CI was reworked to use Nix. As part of that effort, the project-distributed semi-static binary was replaced with an appimage. In this post I dive into an issue I had to debug. The problem The first thing I did after building the appimage was try to run it. ./result --help worked as expected. sudo bpftrace…
Revamping bpftrace’s CI with Nix Home Revamping bpftrace’s CI with Nix I’ve been rewriting the bpftrace CI on-and-off the past few months. Now that it’s nearly done, I thought it would be nice to sum up some of the changes. Previous state The previous iteration of the bpftrace CI relied heavily on Github Actions and docker. Specifically, we had three major workflows: CI, embedded, and code…
bpfilter is forever Home bpfilter is forever This week it was brought to my attention that bpfilter might be delaying our kernel boot sequence. The initial thought was that bpfilter’s usermode upcalls were stalled for some reason and caused boot time stalls. While it is true that module initialization for built-in modules (ie. CONFIG_FOO=y ) is serialized and that in theory it is possible for the…
Kernel string overcopying Home Kernel string overcopying While on a long roadtrip the past weekend, I somehow remembered one of the fun bugs I fixed in the kernel a couple years ago. A bug was reported to me about bpftrace: somehow there were duplicate map entries for the same string key. So something like: # bpftrace -e '... END { print(@) }' ^C @[asdf]: 1 @[asdf]: 2 At first glance, this seemed…
Sudo and signal propagation Home Sudo and signal propagation This week I spent some time debugging a curious case where signals were not being delivered to a sudo ’ed process. Consider this shell script: #!/bin/bash sudo timeout 5 dd if=/dev/urandom of=/dev/null & sudo_pid=$! sudo kill $sudo_pid if ps -p $sudo_pid &>/dev/null; then echo Still running else echo Dead fi What should be printed? If…
The case for a community maintained BPF book Home The case for a community maintained BPF book It’s widely accepted that it’s hard to get started with eBPF (BPF). It’s an extremely fast moving ecosystem and the state-of-the-art today is not the same as it was the last year. For new and perhaps more casual developers who cannot afford to follow along closely, this is a problem. How do we expect…
vmtest: Run your tests in virtual machines Home vmtest: Run your tests in virtual machines Motivation Systems software that depend on specific kernel or host properties often contains logic to gate functionality based on what features are available at runtime. This necessarily begs the question: how do you test platform specific logic? Mocks can help, but they come with their own set of tradeoffs.…
Not so perfect forwarding Home Not so perfect forwarding Consider the following C++ structs: struct Foo { char b; int a; } __attribute__((packed)); struct Bar { Bar(int a) {} }; Now consider the following code: #include <memory> int main() { Foo f; auto p = std::make_unique<Bar>(f.a); } It should compile right? Wrong: /tmp/cppsh-NLnGU.cpp: In function ‘int main()’: /tmp/cppsh-NLnGU.cpp:14:36:…
Visualizing below using Grafana Home Visualizing below using Grafana The fancy way to describe below is “a time traveling resource monitor for modern Linux systems”. The less fancy description is that below is pretty much like top or htop but can persist historical data to disk. It comes with quite a few more neat features but I’ll defer the full explanation and demo to the previous link. Despite…
Generating kernel patchsets Home Generating kernel patchsets Background The Linux kernel famously has an email workflow . Ignoring whether or not it’s a good thing, it is still a fact that it can be tricky to follow the process. One problem I’ve had for years is wrangling git-format-patch and git-send-email to do the right thing. Any time I’ve wanted to send a patchset, I’ve had to search through…
3D modeling with OpenSCAD Home 3D modeling with OpenSCAD The Boring Background A few months ago I got tired of commuting to the local hacker space so I bought a cheap 3D printer to use at home. Like any good neophyte, I printed quite a few precooked models before getting the itch to design some of my own. The biggest hurdle was that I couldn’t find good modeling software to run on Linux. I tried…
File capabilities and /proc/<pid> Home File capabilities and /proc/<pid> I’ve spent the past few evenings staring at a rather interesting kernel “bug”. I put “bug” in quotes b/c, spoiler alert, it’s not a bug. It all started with an integration test failing due to a -EACCES return code. Normally I’d assume it’s a bug with the code under test, but we also happen to cover the same codepath in unit…
BPF and go pprof Home BPF and go pprof Earlier this week I was made aware of what appeared to be an infinite loop in the in-kernel BPF verifier. Golang stacks were pulled, flamegraphs were captured (shoutout to profiler.firefox.com ), and by all appearances it seemed like there was indeed an infinite loop. I spent some time studying the verifier source and nothing really stood out to me as…
Pull request review (prr) Home Pull request review (prr) Last month I developed a new tool that brings mailing-list style code reviews to github. The tool was born out of frustration with github’s point and click web UI. Since I do a lot of code review, the tabbing between my text editor and my browser became a constant source of friction. Ideally, I wanted to stay in my text editor so that my…
Rewriting clipd Home Rewriting clipd clipd is a distributed clipboard I wrote back in 2017. Distributed in this context means that a server somewhere runs the clipd server and accepts requests from clients to either: Store new data into the clipboard, or Grab the current data in the clipboard This solved an annoying problem I had back then where I had too many computers (still do, actually) and…
Ahead-Of-Time compiled bpftrace programs Home Ahead-Of-Time compiled bpftrace programs This page serves as a design document for bpftrace AOT compilation support. Design is currently a work-in-progress and will be (somewhat) regularly updated. Background AOT is motivated by the following technical concerns: Shipping/running LLVM onto prod hosts is resource-heavy (binary size, memory consumption)…
DWARF stack walks and fast symbolization for BPF Home DWARF stack walks and fast symbolization for BPF 10/12/21 update: I have no immediate plans to work on this b/c I’ve recently switched jobs. If you’re really dying to see this implemented maybe we can work something out. Just email me. Background In this document we are proposing solutions for two separate but related problems: stack walking…
Debug Home Debug This page is used for debugging.
Switching to Caddy Home Switching to Caddy I decided to upgrade my ancient Fedora 25 VPS to Fedora 33 this weekend. I couldn’t get the in-place upgrade to work across so many releases so I just blew away the entire server and started fresh. While typing away, I remembered I had heard some good stuff about Caddy . I was also getting annoyed manually refreshing my Let’s Encrypt certs every three…
C++ patterns: static registration Home C++ patterns: static registration Plugin architectures are useful: it’s hard to predict future needs in the present. Thus, it’s often useful to punt as much business logic as possible to the future. But how do we create a robust and scalable plugin architecture? A naive approach is a large if-block. Suppose you have a configuration file where the user…
libbpf-rs: eBPF for the Rust ecosystem Home libbpf-rs: eBPF for the Rust ecosystem Summary libbpf-rs provides safe Rust bindings to libbpf as well as a cargo plugin to help you develop “Compile Once - Run Everywhere” eBPF applications. Use libbpf-rs over BCC to produce smaller, more efficient, and more predictable applications. Background eBPF (BPF) programs have traditionally been written using…
Understanding btrfs internals part 4 Home Understanding btrfs internals part 4 This is the fourth of a multipart series that explains the basics of btrfs ’s on-disk format. At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls. Example code is available here .…
Understanding btrfs internals part 5 Home Understanding btrfs internals part 5 This is the fifth and final part of a multipart series that explains the basics of btrfs ’s on-disk format. At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls. Example code is…
Understanding btrfs internals part 3 Home Understanding btrfs internals part 3 This is the third of a multipart series that explains the basics of btrfs ’s on-disk format. At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls. Example code is available here .…
Understanding btrfs internals part 2 Home Understanding btrfs internals part 2 This is the second of a multipart series that explains the basics of btrfs ’s on-disk format. At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls. Example code is available here .…
Waiting on process exit Home Waiting on process exit How does a process wait for another process to exit? Simple question, right? I’ve been recently working on bpftrace and I had to find an answer for this problem. Ideally, the solution would not do any polling. First, let’s try the most naive solution: waitpid(2) : waitpid(2) #include <iostream> #include <cstdlib> #include <thread> #include…
On writing unmaintainable code Home On writing unmaintainable code The internet’s beaten to death the concept of legacy code , unmaintainable code , bad code , and insert-word-here-code. It’s safe to say nobody really likes dealing with code that doesn’t make much sense. But how about the other side of the coin? Consider this: you’re tasked with writing inherently tricky code. You know this code…
Setting up a barebones website Home Setting up a barebones website I had a very specific workflow in mind when I set out to host a website. First, I had no desire to be locked into a specific platform. I’ve experimented with Github pages but it’s pretty locked into Github’s ecosystem. Second, publication and hosting had to be as cross platform as possible. I wanted to be able to write and publish…
Understanding btrfs internals Home Understanding btrfs internals This is the first of a multipart series that explains the basics of btrfs ’s on-disk format. At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls. Example code is available here . Background Before…
GDB hacking part 2 Home GDB hacking part 2 One of the issues I’ve had with GDB is reliably building the code. The project apparently checks in automake-1.15 artifacts and my host has automake-1.16, so that means any code changes that trigger automake regeneration causes failures. Arch linux doesn’t package auotmake-1.15 like some other distros so I’ve had to get clever. My solution was to wrap the…
Playing with C++ templates Home Playing with C++ templates Last week I found myself quite sick. Rather than wallow in it, I played around with C++ templates. Until now, I’ve only parameterized classes into generic functions. As a daily user of folly , I’ve spent quite some time puzzling over C++ template magic. It’s time I got my feet wet. Simple example We want a generic function to call a…
GDB hacking Home GDB hacking This past weekend I spent some time hacking on GDB where the overall goal was to improve rust debugging support. I considered hacking on LLDB but their upstream stance on language support is fairly conservative . The current state is that they want non C-family languages to be maintained out of tree. Anyone who’s done open source software maintenance knows that’s an…
/proc/[pid] and the missing threads Home /proc/[pid] and the missing threads Suppose you run the following program: use std:: time ; use std:: thread ; fn main() { println! ( "My TID: {}" , nix::unistd:: gettid()) ; let child = thread:: spawn( move || { println! ( "My thread's TID: {}" , nix::unistd:: gettid()) ; loop { thread:: sleep( time::Duration:: from_secs( 1 )) ; } } ) ; child . join() .…
Comparing interned strings Home Comparing interned strings C/C++ has this interesting property where if you define a string literal, the compiler will make sure to only keep one copy of it. This makes sense as it reduces how much static storage is required. For example, if you have: const char* s = "asdf"; const char* s2 = "asdf"; the compiler will make sure “asdf” is stored once in the resulting…
std::vector and emplace_back Home std::vector and emplace_back There are some interesting bits about the C++ runtime. Consider this problem: you want to add an element to a vector of type std::vector<std::pair<int, Foo>> with as little overhead as possible. That means one construction – no extra moves, no extra copies, nothing. For this problem, let us use this implementation of Foo: class Foo {…
systemd and sd_notify(3) Home systemd and sd_notify(3) A little bit of background first: I’ve been spending a lot of time recently mucking around with systemd and its process management features. The latest project I was working on involved implementing hot upgrades on a running server. This means that when used, clients of said server don’t experience any disruption other than a very small…
Creating a basic Linux kernel development setup Home Creating a basic Linux kernel development setup Linux kernel development is something I’ve been wanting to get into for a long time now. I’ve tried and failed to get started over the years, but I finally think I’m in a place where I can do some real kernel work. But before we can dive into kernel development, we have to have a working…
Tracking down a race condition in the Apache Trafficserver logging subsystem Home Tracking down a race condition in the Apache Trafficserver logging subsystem TODO: make this post more coherent Lately at work, I’ve been tracking down a log corruption issue inside of Trafficserver’s logging subsystem . The issue is that there’s often random data inserted into the middle of log statements.…