Recent content on GrantSeltzer
I try to follow the bpf-next mailing list closely, but I always miss things. LWN is incredible, I’m a long time subscriber, but there’s still gaps I’d like to fill. So I built a small automated system to summarize the mailing list for me and publish the results here on my site . It works in two stages. First, a GitHub Actions workflow runs every morning at 5am ET. It queries the…
You can find all the code displayed in this post in my repo here . The proc file system provides to users information about the processes on a running linux system via a file system interface. The directory structure is organized such that every process has its own directory. The files in the directory are useful for finding out permissions, state, open files, and more. I wrote a small post about…
The bpf verifier is tasked with ensuring that bpf programs that are loaded will predictably halt and run safely. It follows all branches of bpf instructions, counting each instruction as another permutation of the program’s state. As it counts, there’s a limit of 1 million instructions. If you’ve been writing bpf programs for a long time or have been looking at outdated examples…
In a couple of my posts from a few years ago ( 1 , 2 ) I explored the idea of attaching bpf programs to Go functions via uprobes. The second post dived into how to extract values of parameters. This can be seen as part 3 of the series as I’m going to demonstrate how to get a stack trace from bpf code. The work described in this post is in contribution to my work at Datadog on the Dynamic…
I recently ran into a hard to find bug in my Go code when using slices. Since I’ve been using Go for over 8 years now, I figured this is something that others will run into and want to share what I had forgotten about how slices work. Take a look at the following Go code: x := []int{1,2,3} y := append([]int{}, x...) z := x We have three separate slices. What I want to briefly explore is the…
BPF has a helper function called bpf_get_attach_cookie . It’s available in bpf programs like kprobes, uprobes, and tracepoints. It can be very useful for many applications so i’ll be exploring how to use it, and going over a few examples from real code i’ve written. A cookie is just an unsigned 64-bit integer. That’s it. You can assign a cookie to a bpf program when you…
There are times when developing your BPF projects when you need to ensure safe access to shared memory. You may have a counter that you’re updating from various different BPF programs and reading in userspace. You may also be updating map values from both userspace and from BPF. In this post i’ll demonstrate a couple different techniques for handing these scenarios safely. Per-cpu maps…
One of the main projects that I work on is libbpfgo . This is a Go wrapper around libbpf , a userspace library for dealing with bpf objects. The wrapper uses CGO to reference individual functions and data types in libbpf, which is written in C. For example, BPFLoadObject() is a libbpfgo API function which calls bpf_object__load() , a libbpf API function. func (m *Module) BPFLoadObject() error {…
The Linux kernel is the biggest open source project in existence. It has tens of thousands of contributors from all around the world, from many different companies and communities. Linux development does not happen in a single github repository, and likely never will. Instead there are many forks of the linux kernel. Development happens in these forks for very specific subsystems, only a fraction…
libbpf is the standard implementation of the userspace library for loading and interacting with bpf programs. It defines the bpf ELF file format and CO-RE features. It’s a crucial part of the bpf ecosystem and is soon approaching its first major release. With this in mind, it’s important to have proper documentation. Introducing libbpf.readthedocs.org . By going to the API docs you can…
Listen on Changelog.com
Skip to 3:11:00 for my talk (or watch the whole thing, it was a great conference!)
If you’re used to developing software in a high level language like Go or Java you likely haven’t thought about supporting different operating system versions. Sure you want to be able to run on Linux, Macos, or Windows, but that’s abstracted away for you. When developing eBPF software you must think about hundreds of Linux kernel versions. When your eBPF programs are reading in…
In my previous posts on the subject of bpf I used a project called BCC to compile, load, and interact with my bpf programs. I, and many other developers, have recently heard about a better way to build ebpf projects called libbpf . There are a few good resources to use when developing libbpf based programs but getting started can still be a quite overwhelming. The goal of this post is to provide a…
A version of this post was also uploaded to Aqua's blog here If you’ve been reading much bpf code recently, you’ve probably seen this: #include 'vmlinux.h' vmlinux.h is generated code. it contains all of the type definitions that your running Linux kernel uses in it’s own source code. This is an important concept to wrap your head around so let me explain. When you build Linux…
In my very first blog post, Dissecting Go Binaries , I began to explore ELF files. That is the default format of executable binaries that Go produces on unix-like operating systems like Linux and MacOS. My most recent project, weaver , has me exploring a related object file format that Go leverages, DWARF . The DWARF specification is, of course, not Go specific. The DWARF object file specification…
Code can be found here
Listen on Changelog.com
In part 1 of this series we learned about how to attach uprobes and eBPF programs to specific functions in Go programs. We went through an example where we attached our probe to the handler of a webserver. Everytime the probe was triggered we simply printed out a log saying that the handler was called. This could be adapted to record metrics, perhaps using counters instead of log lines. In this…
In the past year I’ve decided that I haven’t perpetuated enough Brooklyn stereotypes and have taken up fermentation as a hobby. This post goes through a few of the experiments that I’ve run with varying success. My set up is pretty basic. I have a small cabinet/closet in the corner of my kitchen that i’ve designated as the ‘fermentation station’ . I have a…
eBPF is a virtual machine, similar in concept to the JVM, except it’s inside the Linux kernel. It lets you write C-like code, compile it, and load the byte code into the kernel. You can then attach hooks to your loaded eBPF program to trigger it to run. Those hooks could be things like system calls, kprobes , or uprobes . There’s many use cases for eBPF. Considering eBPF programs have…
Karn aims to provide for Linux what entitlements provide for iOS, or what pledge provides for OpenBSD. It does this by translating a high level set of intuitive ’entitlements’ into complex seccomp profiles. For example, A developer using Karn can simply specify that their application needs to make network connections, or exec other processes and karn will handle granting it permission…
I love the procfs because it provides an intuitive and easy to use interface for interacting with the kernel. In the same way you’d normally use the command line for exploring files, you can control and find out information about currently running processes. All without the need for system calls. A procfs is mounted at /proc by default. If you go into /proc you’ll see something like…
Remember back to when your teacher in grade school would write on sheets of clear plastic, stack them, and display them with an overhead projector? That is exactly how overlayfs works… except with filesystems. Overlay filesystems allow you to take multiple directory trees and view them as if they were merged together. The added benefit is that changes in the merged view do not affect the…
This post discusses building go test binaries and walking through them with delve I recently was working on debugging a unit test I wrote in Go. I couldn’t figure out why one of my test cases was causing a runtime error that never happened when running my actual program. I was using a runtime directive so I suspected there may be some difference between doing a go test and a go run . I was…
In this post we discuss how standardized streams (`stdin`/`stdout`/`stderr`) work on Linux, especially related to shells. Every well known operating system has a concept of standardized streams. These consist of standard input, output, and error. As their name purports, they are the standard places for proccesses to read input, and send output. They’re a crucial concept that allows various…
This post highlights a linker directive in Go. It allows us to import functions from a dynamic library even when CGO is disabled. I use the example of a contribution I recently made to the net and runtime packages to demonstrate its use case. Go has many little known features that allow you to give instructions to the compiler, linker, and other parts of the toolchain using special comments. Dave…
In this post we explore best practices of defining interfaces in Go. We use `io.Writer` to break down patterns and antipatterns. A perfect example of a properly designed Go interface is io.Writer : type Writer interface { Write(p []byte) (n int, err error) } It is short, simple, and powerful. The Write method takes a very generic slice of bytes and writes it to something . In Rob Pike’s Go…
This post will introduce you to cgroups. The goal is to give a comprehensive enough explanation of cgroups and subsystems to broadly understand what they accomplish and how. The major focus will be on the CPU cgroup. The [second section](#using-cgroups) and [third section](#practically-using-cgroups) will explore using cgroups from the command line. Control groups (or cgroups) are a feature of the…
This post will introduce you to the basic concepts of disassembly and walk through writing a simple disassembler in Go. The goal is to make the whole concept of disassembly as accessible as possible and hopefully give you, the reader, a takeaway or two about how compiled Go binaries work. Assembly code cannot lie. By reading assembly we become as close as possible to knowing what’s being run on…
Bialys Bialys are a traditional roll that are fundamental to Ashkenazi Jewsish cuisine. They were a staple at bakeries in New York throughout the 20th century but are hard to find outside of areas with large Ashkenazi populations. Bagels simply get all the glory. Whereas bagels are supposed to be tough and chewy, Bialys are soft, light, and have an even airy crumb structure. Instead of having a…
Miami rolls + Challah Miami rolls are a challah based onion pocket roll. They’re very soft, sweet, flavorful, and delicious. Pretty traditional to jewish bakeries in New York. I worked at a bakery all throughout high school and these were my favorite things there. When they closed down in 2020 I tried to get the recipe but the owner wouldn’t share it with me. Through 15+ iterations I…
Recipes: Miami Rolls Bialys Split Pea Soup
Split Pea Soup Ingredients 2 cups green split peas 7 cups vegetable broth 1 cup coconut milk 1 cup dried potato flakes 2 large yellow onions 2 tbs minced garlic 2 tbs diced fresh ginger 1 bunch of parsely 3 tbs olive oil 1 tbs herbs de provence 2 tbs curry herb blend 1 tbs salt 2 tbs carraway seed Instructions Chop the yellow onions. Mince the garlic. Dice the ginger. Chop the parsely.
My name is Grant Seltzer Richman. I currently work at Datadog. I have worked as a software engineer at Aqua , Oscar Health , Capsule8 , and Red Hat . My interests include systems programming, cycling, triathlon, security, video games, baseball, and reading. I mostly will be writing blog posts about linux, security, and various hobbies. Feel free to reach out if you want to say hi!
A quick tool I use for birding. If I’m searching for a specific species I use this to see where other birders have reported sightings on eBird . It gives all checklists within the specified parameters. Try it yourself! Get an eBird API Token . Enter your region, look for the bird of your dreams! Birding is fun!