Recent content on 0x74696d
Dormant Last read · last published · next check
Read 4 days ago and current, but nothing has been published for 2 years.
Today I want to start by talking about hardware signals and end with an explanation of why Nomad doesn't implement OpenTelemetry. The terms edge-triggered and level-triggered come from signal processing for computer processors. Very roughly, for the CPU to service I/O or other changes in external state, it needs to receive an interrupt (or "trap") that tells it to stop whatever it's currently…
In a 1999 episode of the Simpsons, Homer briefly changes his name to Max Power in a bumbling attempt to adopt a more successful persona via nominative determinism. The episode contains this memorable exchange: Homer: "Kids, there's three ways to do things. The right way, the wrong way, and the Max Power way!" Bart: "Isn't that the wrong way?' Homer: "Yeah, but faster!" That pretty much sums up my…
The house we rent has high ceilings along with lots of ceiling fans, which is great for the warm months here in Philadelphia. Most of these are the pull-cord type: a wall switch powers it on or off and then there are two pulls that toggle the lamp and set the fan speed. But the one above our dining room tables has a remote control, and it's a good illustration of two different kinds of UX blunder.…
Recent Linux kernels have the kqueue-alike io_uring interface for asynchronous I/O. Instead of making read and write syscalls, you write batches of I/O requests to a circular buffer in userland called the submission queue, and make a io_uring_enter syscall to submit them to the kernel. Instead of making individual syscalls, io_uring submission queue entries (SQEs) take an opcode for the specific…
The email comes that the newly-formed Platform Team will be migrating all two thousand of the company's services to the New Platform TM . The new platform will ensure there's a single workflow across the company for building, testing, securing, and shipping software. The old legacy cruft that was weighing down delivery will be swept away. Maybe there's an accompanying slide show at the all-hands…
If I go back far enough, it's clear my professional career in tech started with the realization that AutoCAD's command line interface was in fact a shell . It didn't just accept instructions, but you could write little programs in this "weird language" called Lisp and it would make calculations based on your drawing or even make changes to the drawing 1 . I started with automating repetitive work,…
Folks are rightfully excited about the performance benefits to Go 1.17's changes to function calling convention, but I was a little disappointed to discover it doesn't make BPF uretprobe s possible. As it turns out, I hadn't fully considered how Go's relocatable stacks complicate the situation. One of the advantages of Go's fast compile times is that "printf debugging" gets you pretty far. You…
Several of my personal projects require creating and destroying a lot of virtual machines, so I've started porting them over from QEMU to Firecracker to speed up development. I'm now at the point where I need to make some of these VMs accessible from the internet and other machines on the LAN. My scenario is that I have three physical machines (2 NUCs and my development laptop) that act as virtual…
It was before lunch on my first day when I found myself in a conference room during a total service outage, with thirty or so other men yelling over each other 1 . This was particularly alarming because during my interviews we'd talked a lot about building disciplined incident response culture, developing deep insight about the platform, and avoiding normalization of deviance. Clearly, I'd…
At my current gig and several before that, the initial engineering design document is the Request for Comments (RFC), sometimes called the Request for Discussion (RFD). If you've been reading the series on building a ZFS plugin for Nomad 1 , you might have asked yourself if this kind of stumbling through the design is typical of the RFC documents I've written. But that series is really about all…
There's an enormous wealth of data analysis and visualization tools available, from full-fledged managed services like Honeycomb all the way down to Python libraries like Matplotlib. These days I'm writing shrink-wrapped infrastructure software, so when I'm debugging problems I've been leaning way more to one end of that spectrum: writing single purpose tools. I can just barely fool myself into…
A particularly annoying bug in Hugo that I've been running into is that it will output 1-byte files for the index and other pages. This is not sparking joy. What's particularly bad about it is that it's not at all consistent. Any given build will just randomly decide that the front page or the RSS feed should be empty. So every time I pushed to Netlify it'd be a crap shoot as to whether or not the…
A short and mostly unserious rant. Much of the industry seems to have come around to changing the default branch for git from master to main . It's absolutely terrible. Oh, not changing from master . That's fine. A modest improvement in making our industry kinder. I'm all for it. But we picked main . Literally any time anything to do with git comes up, you've got a whole mess of people complaining…
At some point I noticed that some large portion of containers I've seen have some kind of start.sh script file doing some setup and then calling the actual application. Unfortunately a ton of these break features of the application server. Like the previous post on dropping signals, the way this typically manifests is the application server can't reload configuration or gracefully shut down. Your…
A lot of go applications try to do something clever with signals and end up dropping signals on the floor. I've definitely written this kind of bug myself. It's not a community practice to lean on an application server rather than the stdlib, so that creates an opportunity for folks to incorrectly implement it from scratch. Note that we're not talking about signal-safety(7) . For purposes of this…
Recently The Register published an interview with Microsoft's Sarah Novotny where she claimed that the Linux kernel project's reliance on plain-text email was a barrier to entry for new kernel developers. Predictably a bunch of folks showed up on Twitter to heap abuse and gatekeep people's email clients, and just as predictably a lot of well-meaning folks took the opposing side that because those…
On Memorial Day weekend I grilled a steak, as one does. 1 A few days before, I took it out of the freezer and put it into the fridge to thaw on a plate. Late Sunday afternoon I took it out of the fridge and patted it down with a paper towel. Then I put it back in the fridge. Then I took it out and rubbed it down with oil and salt and pepper. Then I put it back in the fridge. I stepped outside to…
Mise-en-place is the religion of all good line cooks. Do not fuck with a line cook's 'meez' — meaning his setup, his carefully arranged supplies of sea salt, rough-cracked pepper, softened butter, cooking oil, wine, backups, and so on. — Anthony Bourdain, Kitchen Confidential photo by Charles Haynes - https://www.flickr.com/photos/haynes/500435491 , CC BY-SA 2.0,…
The conversation goes something like this: Them: "Our service can't be autoscaled, run on spot instances, or have its host restarted at random because it runs long-running tasks." Me: "Are the tasks idempotent?" Them: "No, but they're checkpointed." Me: "Even if we don't autoscale, run on spot instances, or ever update the host, the host can randomly fail at any time." Them: "Yes, but that's less…
We all figure out our first Python bugs by sprinkling some print statements over our code. As we gain experience, our debugging toolbox becomes richer and we can figure out harder bugs in development. But production systems provide a different kind of challenge, and this challenge is amplified when we try to debug in a containerized environment. We need to be able to debug running code safely,…
![](https://blog.0x74696d.com/posts/tracing-optimized-edge-triggered-systems/"https://blog.0x74696d.com/images/20240323/signals.svg")
![](https://blog.0x74696d.com/posts/two-kinds-of-ux-blunder/"https://blog.0x74696d.com/images/20221127/remote.jpg")
![](https://blog.0x74696d.com/posts/challenges-of-bpf-tracing-go/"https://blog.0x74696d.com/images/20210828/stack.svg")
![](https://blog.0x74696d.com/posts/single-purpose-visualization/"https://blog.0x74696d.com/images/20210130/metrics.svg")
![](https://blog.0x74696d.com/posts/mise-en-place/"https://blog.0x74696d.com/images/20190604/1200px-Mise_en_place_for_hot_station.jpg")
![](https://blog.0x74696d.com/posts/checkpointing-failure/"https://blog.0x74696d.com/images/20190217/plot.png")
![](https://blog.0x74696d.com/posts/debugging-python-containers-in-production/"https://blog.0x74696d.com/images/20180503/eBPF-diagram.png")