Software and articles by John Millikin.
Rust and dynamically-sized thin pointers One of Rust's notable differences from C is its requirement that all values have a defined size, which enables runtime bounds-checking and advanced static analysis tooling such as MIRI . For dynamically-sized types (DSTs) this requirement is implemented using thick pointers, such that each pointer to a dynamically-sized value is an (address, size) tuple.…
vu128: Efficient variable-length integers Why variable-length integers? The basic goal of a variable-length integer encoding is to allow small numbers to be encoded in fewer bytes than larger numbers. Among other uses, they're popular for TLV formats such as Protocol Buffers , which contain lots of small numbers (tags) and have a design goal of space efficiency. The two encodings with…
Creating TUN/TAP interfaces in Linux The basic approach to writing a TUN/TAP client (such as a VPN) for Linux is: Open the /dev/net/tun device as a file, which (once configured) will communicate network traffic to userspace. Allocate (or bind) a virtual network interface with the file handle using ioctl(TUNSETIFF) . Configure the network interface's address and link state. Process network traffic…
Running SunOS 4 in QEMU (SPARC) SunOS is a historical UNIX operating system widely used from the mid 80s into the early/mid 90s. Older versions of QEMU struggled to emulate the SPARC platform that SunOS ran on, but QEMU v7.2 supports SPARC well enough to install and run SunOS without any unusual workarounds. Installation media The installation CD-ROM for SunOS 4.1.4 (also branded Solaris 1.1.2) is…
Improved UNIX socket networking in QEMU 7.2 QEMU 7.2 quietly introduced two new network backends, -netdev dgram and -netdev stream . Unlike the older -netdev socket , these new backends directly support AF_UNIX socket addresses without the need for an intermediate wrapper tool. The situation up until now QEMU has a -netdev socket network backend, which will send/receive Ethernet frames via TCP…
Debugging Win32 binaries in Ghidra via Wine Ghidra is a cross-platform reverse-engineering and binary analysis tool, with recent versions including support for dynamic analysis. I want to try using it as a replacement for IDA Pro in reverse-engineering of Win32 binaries, but hit bugs related to address space detection when running gdbserver with Wine ( ghidra#4534 ). This post contains custom GDB…
Running BeOS 5 in QEMU (i386) BeOS is an operating system from the '90s, notable for its prescient technical decisions and abject business failure [ 1 ] . It embraced multi-threading at a time when 100mhz CPUs powered top-shelf workstations, and featured metadata-backed virtual folders ten years before their arrival in mainstream OSes. Installation media The installation CD-ROM for BeOS Pro…
Gmail accepts forged YouTube emails This morning I woke up to an official-looking email from YouTube in my inbox, addressed to an address that isn't mine. 
 Long ago this sort of thing would happen if someone sent an email with forged headers [ 1 ] (e.g. to fish for logins), but the advent of DKIM and DMARC has relegated header forging to ancient history. I was greatly surprised to see that…
Compacting Lunr search indices Lunr is a small JavaScript library for full-text search, which I recently used to implement client-side search for this site. The user experience of client-side search depends in part on how large the search index is, and Lunr's default JSON encoding is more verbose than it needs to be. This page describes a more compact encoding that can reduce the serialized index…
JSON is not a YAML subset People on the internet believe that JSON is a subset of YAML, and that it's safe to parse JSON using a YAML parser: 
 
 Following this advice will end badly because JSON is not a subset of YAML. It is easy to construct JSON documents that (1) fail to parse as YAML, or (2) parse to valid but semantically different YAML. The second case is more dangerous because…
Stateless Kubernetes overlay networks with IPv6 The Kubernetes network model is typically implemented by an overlay network, which allows pods to have an IP address decoupled from the underlying fabric. There's dozens of different overlay network implementations that combine a stateful IPv4 address allocator with VXLAN as a transport layer. IPv4 overlay networks have a number of well-documented…
Extending VSCode with WebAssembly Two years ago I filed Microsoft/vscode#65559 asking for WebAssembly support in VSCode extensions. At the time, WASM was supported by Node.JS but the WebAssembly symbol wasn't available in the extension's evaluation scope. That issue didn't get much activity from upstream but the other day I tried it again, and … it worked! Below is a small 'hello world' LSP-based…
Notes on cross-compiling Rust One of my current hobby projects involves running Rust binaries on a Raspberry Pi. There are three computers involved: the Pi itself (ARMv7 Linux), my desktop (x86-64 Linux), and sometimes my laptop (x86-64 macOS). The release of Cyberpunk 2077 means that my desktop will be spending more time booted into Windows, so I needed to figure out how to get the macOS machine…
First impressions of Rust I've been wanting to write a big project in Rust for a while as a learning exercise, and actually started one in late 2018 (a FUSE server implementation). But then life happened and I got busy and never went anywhere with it. Due to certain world circumstances I'm currently spending a lot of time indoors so rust-fuse ( docs ) now exists and is good enough to write basic…
Commentary on “Stop Using Encrypted Email” Latacora’s recent article Stop Using Encrypted Email prompted a lot of comments on Tildes and Hacker News , which is not surprising considering the author’s blunt approach to a delicate topic. I agree with its recommendations and also think it would be good if other people did so, thus, I’m going to try expanding on it a little. Think of this as comments…
By any other CNAME When an engineer joins Google, they are issued a workstation – a physical computer in tower [ 1 ] form-factor that sits under their desk. Workstations have names, which the engineer gets to choose, and the fully-qualified hostname consists of this name plus an office-specific suffix. Workstations in Mountain View are in .mtv.corp.google.com , workstations in New York are…
SRE School: No Haunted Forests Engineer debugging a Puppet manifest (2018, colorized) All industrial codebases contain bad code. To err is human, and situations get very human when you're staring down the barrel of a launch deadline. You've heard the euphemism tech debt , where like a car loan you hold a recurring obligation in exchange for immediate liquidity. But this is misleading: bad code is…
(More) Effective Go Unbounded Iteration 'Unbounded iteration' is when you need to iterate over a sequence without knowing its total length. For example, receiving rows from a database query or data chunks from an HTTP response. Other languages have a native concept of 'iterators' such that iterating over an array and stream use the same syntax, but Go doesn't do this. The best approach to…
Error Beneath the WAVs This is a follow-up to Why I Ripped The Same CD 300 Times . By the end of that page I'd identified a fragment of audio data that could cause read errors even if it was isolated and burned to a fresh CD. Further testing yielded a 'cursed WAV' that consistently prevents perfect rips on different brands of optical drive, ripping software, and operating system. EDIT…
Why I Ripped The Same CD 300 Times I collect music by buying physical CDs, digitizing them with Exact Audio Copy , and scanning the artwork. This is sometimes challenging if the CD was self-published in a limited run in a foreign country ten years ago. It is very challenging if the CDs have an innate defect that renders some tracks unreadable. (Русский перевод: Зачем я рипнул один компакт-диск 300…
Effective gRPC This page documents habits and styles I've found useful when working with gRPC and Protocol Buffers . gRPC Error Reporting Use the google.protobuf.Status message to report errors back to clients – this type should be special-cased by the gRPC library for your language (e.g. grpc-go has 'google.golang.org/grpc/status' . This message can contain arbitrary sub-messages, so…
Bazel School: Toolchains I've recently been using Bazel as a multi-platform distributed build system. Bazel itself supports this pretty well, but many of the user-contributed extension libraries don't make good use of Bazel's toolchains and therefore break when multiple OSes are involved in a build. I hope the situation can be improved by documenting nascent best practices. This page is a bit…
Mojibake in Surugaya Javascript Yesterday I bought some used CDs from the online store Surugaya . The checkout process was broken in an interesting way: when I clicked the payment method confirmation button, nothing happened. I switched from Chrome to Firefox and was able to place an order successfully [ 1 ] . The bug A quick look in the web console showed some errors in loading the page's…
UNIX Syscalls On UNIX-like operating systems, userland processes invoke kernel procedures using the 'syscall' feature. Each syscall is identified by a 'syscall number' and has a short list of parameters, which both can vary between operating systems, hardware platforms, and configuration options. Performing a syscall is usually done via a special assembly instruction, though some platforms use…
SRE School: Health Checking Any service that has complex logic or external dependencies might stop working for unexpected reasons. While instrumentation and monitoring can help bring these problems to human attention, it can be difficult to use dashboards or alerts for low-latency automated responses. A load balancer, for example, should respond to unhealthy backends on the order of seconds…
Reddit Front Page (2018) Over the past few months I've noticed I get a lot less enjoyment out of browsing Reddit. There wasn't any clear reason, just a general feeling that every hour I spent there was an hour wasted. It didn't use to be that way, I think – while there were always some forums there filled with noise, it also contained fresh analyses and insightful commentary and regularly…
Re:Creators Episode 21 Episode 21 of Re:Creators ends with a rather nicely typeset message to the viwer, in Japanese and Latin: Mundum divit factum, atque pulchre. 世界は豊かに、そして美しく That's some unusual Latin. I wonder how they translated it? Can I do better? Google Translate [ 1 ] and Yandex Translate both support Latin output, but their results don't match the screencap. Bing doesn't support Latin at…
SRE School: Instrumentation Instrumentation is the foundation of a monitoring infrastructure. It is the part that directly touches the system(s) being monitored, the source of raw data for our collectors and analyzers and dashboards. It is also the only part that is not under an SRE team's direct control – instrumentation is usually plumbed through the codebase by product teams. Given this,…
haskell-cpython: Calling Python libraries from Haskell Haskell's a great language; it's efficient, consistent, terse, reliable, and so on. But if there's one thing Haskell's not, it's 'batteries included'. Compared to popular dynamic languages, such as Python and Ruby, Haskell has a very limited module library. Writing bindings to Python libraries (via the Python/C API ) is an easy and practical…
Monad is not difficult In Haskell, the typeclass Monad is a way for programmers to customize how sequences of function calls are run. Originally the goal was just to make IO-heavy code easier to read, but it turns out that Monad-shaped APIs are wonderfully flexible and can simplify all sorts of programming problems. For example, say we want to define a function to look up two people in a database,…
Understanding Iteratees Iteratees are an abstraction discovered by Oleg Kiselyov, which provide a performant, predictable, and safe alternative to lazy I/O. Though the data types involved are simple, their relationship to incremental processing is not obvious, and existing documentation ranges in quality from merely dense to outright baffling. This article attempts to clarify the concepts and use…