It is not uncommon to meet people in real life who know this blog. Just the other day I got to talking to someone who had overheard me discussing some compiler thing with a colleague and asked if they knew mcyoung.xyz , and when they said the recent Go articles were pretty cool, I mentioned it was my blog. I also know I have a pretty big audience because I see some of my more popular articles…
Let’s talk about linker symbols. In C, when you write out a function signature, like this, you are informing the compiler that a symbol of appropriate ABI will exist at link time. void * malloc ( size_t bytes ); C You can prefix this with extern , but that’s technically not necessary and it irks me a little when people do it. You can also do this with globals, and in this case extern…
Silicon designers are bad at designing secure hardware. Embarrassingly so , sometimes. This means that low-level cryptography, as well as code which directly handles key material, often needs to be written in a particularly delicate style called “constant-time”. “Constant-time” is a bit of a misnomer. It does not mean that the code’s time complexity is O ( 1 )…
If you’ve read anything about compilers in the last two decades or so, you have almost certainly heard of SSA compilers , a popular architecture featured in many optimizing compilers, including ahead-of-time compilers such as LLVM, GCC, Go, CUDA (and various shader compilers), Swift 1 , and MSVC 2 , and just-in-time compilers such as HotSpot C2 3 , V8 4 , SpiderMonkey 5 , LuaJIT, and the…
Go’s interfaces are very funny. Rather than being explicitly implemented, like in Java or Rust, they are simply a collection of methods (a “method set”) that the concrete type must happen to have. This is called structural typing, which is the opposite of nominal typing. Go interfaces are very cute, but this conceptual simplicity leads to a lot of implementation problems (a theme…
Historically I have worked on many projects related to high-performance Protobuf, be that on the C++ runtime, on the Rust runtime, or on integrating UPB , the fastest Protobuf runtime, written by my colleague Josh Haberman . I generally don’t post directly about my current job, but my most recent toy-turned-product is something I’m very excited to write about: hyperpb . Here’s…
It’s no secret that my taste in programming languages is very weird for a programming language ~~enthusiast~~ professional. Several of my last few posts are about Go, broadly regarded as the programming language equivalent of eating plain oatmeal for breakfast. To make up for that, I’m going to write about the programming language equivalent of diluting your morning coffee with…
Most people don’t know that Go has special syntax for directives. Unfortunately, it’s not real syntax, it’s just a comment. For example, //go:noinline causes the next function declaration to never get inlined, which is useful for changing the inlining cost of functions that call it. There are three types of directives: The ones documented in gc ’s doc comment . This…
You’d need a very specialized electron microscope to get down to the level to actually see a single strand of DNA. – Craig Venter TL;DR: buf convert is a powerful tool for examining wire format dumps, by converting them to JSON and using existing JSON analysis tooling. protoscope can be used for lower-level analysis, such debugging messages that have been corrupted. note I’m…
I’ve been very fortunate to dodge a nickname throughout my entire career. I’ve never had one. – Jimmie Johnson TL;DR: Enum values can have aliases. This feature is poorly designed and shouldn’t be used. The ENUM_NO_ALLOW_ALIAS Buf lint rule prevents you from using them by default. note I’m editing a series of best practice pieces on Protobuf, a language that I work on which…
My dad had a guitar but it was acoustic, so I smashed a mirror and glued broken glass to it to make it look more metal. It looked ridiculous! –Max Cavalera TL;DR: Avoid import public and import weak . The Buf lint rules IMPORT_NO_PUBLIC and IMPORT_NO_WEAK enforce this for you by default. note I’m editing a series of best practice pieces on Protobuf, a language that I work on which has…
Bad humor is an evasion of reality; good humor is an acceptance of it. –Malcolm Muggeridge TL;DR: Protobuf’s distributed nature introduces evolution risks that make it hard to fix some types of mistakes. Sometimes the best thing to do is to just let it be. note I’m editing a series of best practice pieces on Protobuf, a language that I work on which has lots of evil…
Smart people learn from their mistakes. But the real sharp ones learn from the mistakes of others. –Brandon Mull TL;DR: enum s inherit some unfortunate behaviors from C++. Use the Buf lint rules ENUM_VALUE_PREFIX and ENUM_ZERO_VALUE_SUFFIX to avoid this problem (they’re part of the DEFAULT category). note I’m editing a series of best practice pieces on Protobuf, a language that I…
Even though I am a C++ programmer at heart, Go fascinates me for none of the reasons you think. Go has made several interesting design decisions: It has virtually no Undefined Behavior 1 . It has very simple GC semantics that they’re mostly stuck with due to design decisions in the surface language. These things mean that despite Go having a GC, it’s possible to do manual memory…
As a matter of fact, when compression technology came along, we thought the future in 1996 was about voice. We got it wrong. It is about voice, video, and data, and that is what we have today on these cell phones. –Steve Buyer TL;DR: Compression is everywhere: CDNs, HTTP servers, even in RPC frameworks like Connect. This pervasiveness means that wire size tradeoffs matter less than they used…
Cross-compiling is taking a computer program and compiling it for a machine that isn’t the one hosting the compilation. Although historically compilers would only compile for the host machine, this is considered an anachronism: all serious native compilers are now cross-compilers. After all, you don’t want to be building your iPhone app on literal iPhone hardware. Many different…
I wake up every morning and grab the morning paper. Then I look at the obituary page. If my name is not on it, I get up. –Ben Franklin TL;DR: Don’t rename fields. Even though there are a slim number of cases where you can get away with it, it’s rarely worth doing, and is a potential source of bugs. note I’m editing a series of best practice pieces on Protobuf, a language that I work on…
Every modern programming language needs a formatter to make your code look pretty and consistent. Formatters are source-transformation tools that parse source code and re-print the resulting AST in some canonical form that normalizes whitespace and optional syntactic constructs. They remove the tedium of matching indentation and brace placement to match a style guide. Go is particularly well-known…
A second post on Go silliness (Sunny, aren’t you a C++ programmer?): in 1.23, Go finally added custom iterators. Now, back when I was at Google and involved in the Go compiler as “the annoying Rust guy who gets lunch with us”, there were proposals suggesting adding something like this, implemented as either an interface or a func: type Iter [ T any ] = func () ( T , bool ) Go…
Lately I’ve been finding myself writing a bit of Go, and I’ve picked up various fun “layout secrets” that help inform how I write code to minimize hidden allocations, and generally be kind to the optimizer. This article is a series of notes on the topic. This post is about Go implementation details, so they can probably break you at any time if you rely on it. On the other…
JSON is extremely popular but deeply flawed. This article discusses the details of JSON’s design, how it’s used (and misused), and how seemingly helpful “human readability” features cause headaches instead. Crucially, you rarely find JSON-based tools (except dedicated tools like jq ) that can safely handle arbitrary JSON documents without a schema—common corner cases can lead to data corruption!…
I will often say that the so-called “C ABI” is a very bad one, and a relatively unimaginative one when it comes to passing complicated types effectively. A lot of people ask me “ok, what would you use instead”, and I just point them to the Go register ABI , but it seems most people have trouble filling in the gaps of what I mean. This article explains what I mean in detail.…
Another explainer on a fun, esoteric topic: optimizing code with SIMD (single instruction multiple data, also sometimes called vectorization ). Designing a good, fast, portable SIMD algorithm is not a simple matter and requires thinking a little bit like a circuit designer. Here’s the mandatory performance benchmark graph to catch your eye. “SIMD” often gets thrown around as a…
Linear algebra is undoubtedly the most useful field in all of algebra. It finds applications in all kinds of science and engineering, like quantum mechanics, graphics programming, and machine learning. It is the “most well-behaved” algebraic theory, in that other abstract algebra topics often try to approximate linear algebra, when possible. For many students, linear algebra means…
I write compilers for fun. I can’t help it. Consequently, I also write a lot of parsers. In systems programming, it’s usually a good idea to try to share memory rather than reuse it, so as such my AST types tend to look like this. pub enum Expr <' src > { Int ( u32 ) Ident ( &' src str ), // ... } Rust Whenever we parse an identifier, rather than copy its name into a fresh String , we…
The other day, I saw this tweet . In it, Andrew Gallant argues that reaching for LLVM IR, instead of assembly, is a useful tool for someone working on performance. Unfortunately, learning material on LLVM is usually aimed at compiler engineers, not generalist working programmers. Now, I’m a compiler engineer, so my answer is of course you should know your optimizer’s IR. But I do think…
Rust and C++ both have very similar operational semantics for their “anonymous function” expressions (they call them “closures” and “lambdas” respectively; I will use these interchangably). Here’s what those expressions look like. auto square = []( int x ) { return x * x ; } C++ let square = | x : i32 | x * x ; Rust The type of square in both versions is…
Traits are the core of polymorphism in Rust. Let’s review: trait Stringer { fn string ( & self ) -> String ; } impl Stringer for i32 { fn string ( & self ) -> String { format! ( '{self}' ) } } // Prints `42`. println! ( '{}' , 42. string ()); godbolt Rust Notice that we call the trait method Stringer::string directly on the value in question. This means that traits (at least, those currently…
Let’s say we’re building an allocator. Good allocators need to serve many threads simultaneously, and as such any lock they take is going to be highly contended. One way to work around this, pioneered by TCMalloc, is to have thread-local caches of blocks (hence, the “TC” - thread cached). Unfortunately threads can be ephemeral, so book-keeping needs to grow dynamically, and…
I’m not really one to brag publicly about expensive toys, but a few weeks ago I managed to get one that’s really something special. It is a Curta Type II , a mechanical digital 1 calculator manufactured in Liechtenstein between the 50s and 70s, before solid-state calculators killed them and the likes of slide-rules. I have wanted one since I was a kid, and I managed to win an eBay…
Let’s talk about C++ templates. C++ is famous for relegating important functionality often built into the language to its standard library 1 . C++11 added a number of very useful class templates intended to make generic programming easier. By far the most complicated is std::tuple<> , which is literally just a tuple. It turns out that implementing std::tuple<> is complicated. Very, very…
Alkyne is a scripting language I built a couple of years ago for generating configuration blobs. Its interpreter is a naive AST walker 1 that uses ARC 2 for memory management, so it’s pretty slow, and I’ve been gradually writing a new evaluation engine for it. This post isn’t about Alkyne itself, that’s for another day. For now, I’d like to write down some notes for…
Almost a year ago I developed the moveit Rust library, which provides primitives for expressing something like C++’s T&& and move constructors while retaining Rust’s so-called “destructive move property”: moving a value transfers ownership, rather than doing a funny copy. In an earlier blogpost I described the theory behind this library and some of the motivation, which I…
A Turing tarpit is a programming language that is Turing-complete but very painful to accomplish anything in. One particularly notable tarpit is Brainfuck , which has a reputation among beginner and intermediate programmers as being unapproachable and only accessible to the most elite programmers hence the name, as Wikipedia puts it: The language’s name is a reference to the slang term…
Low level software usually has lots of .cc or .rs files. Even lower-level software, like your cryptography library, probably has .S containing assembly, my least favorite language for code review. The lowest level software out there, firmware, kernels, and drivers, have one third file type to feed into the toolchain: an .ld file, a “linker script”. The linker script, provided to Clang…
Writing unsafe in Rust usually involves manual management of memory. Although, ideally, we’d like to exclusively use references for this, sometimes the constraints they apply are too strong. This post is a guide on those constraints and how to weaken them for correctness. “Unmanaged” languages, like C++ and Rust, provide pointer types for manipulating memory. These types serve…
I’ve been told I need to write this idea down – I figure this one’s a good enough excuse to start one of them programming blogs. TL;DR You can move-constructors the Rust! It requires a few macros but isn’t much more outlandish than the async pinning state of the art. A prototype of this idea is implemented in my moveit crate. The Interop Problem Rust is the best contender…