Recent content in Blogs on Alex Gaynor
(I work for Anthropic.) One of the most important (and hardest!) questions for a security engineer to answer is “how many vulnerabilities exist in this code base”. You can know how many vulnerabilities you are aware of but haven’t patched yet. You might be able to do some arithmetic and estimate based on historical vulnerability time between introducing and discovery how many…
Anytime you see some someone describing how they used a new vulnerability research technique to find a bug, you’ll see an internet commenter state with confidence that static analysis could have found this bug. This pattern is so consistent as to be a law of physics. However, despite the internet commentariat’s confidence in static analysis, there are still a large number of…
(I’m not a lawyer, and I did precious little research in drafting this. It should be understood as a thought experiment and intellectual provocation. It’s also possible, if not likely, that this is a well-known observation and adds nothing novel.) From time to time, federal courts find that part of a statute is incompatible with the constitution. When this happens, the court must…
I’ve recently created and open sourced a new serialization format, named Relish1, and a Rust library implementing it. They can be found here. I don’t expect many people to adopt it – there are quite a few choices for serialization libraries available to people already – however, it has some nice design properties that I want to describe. Relish is a binary serialization…
When a project receives a vulnerability report, what’s the first question they should ask? I believe the correct answer is that we should ask: In what way does the claimed vulnerability violate our threat model? A lot of times the answer is obvious, we don’t need to spend a lot of time interrogating how SQL injection or a buffer overflow violates our threat model. But it’s not…
(Editor’s Note: My day job is at Anthropic.) This story starts where all good stories start, with ASN.1. ASN.1 is… I guess you’d call it a meta-serialization format? It’s a syntax for describing data abstractly (a notation, you might say), and then there’s a bunch of different actual encodings that you can use to turn data into bytes. There’s only one encoding…
Benchmarks are how we measure performance, obviously. They’re a program that we run to tell us how fast our code is. This intuitive definition is what most of us would say if presented with the title question. But I want to suggest a simple question: if benchmarks tell us how fast something is, what does that make production performance metrics? Try to answer this question without admitting…
Standard libraries are among the most debated topics in programming language design. They are by turns the most widely used APIs in any programming ecosystem and also the most criticized. This post will explore what makes standard libraries what they are. It is entirely non-normative; my goal is to describe what standard libraries are, not specify what they should be. A standard library is the set…
I’ve previously written about common reasons that developing open source software is not funded. That post focused mostly on the individual circumstances and choices of open source maintainers and projects. In contrast, this post attempts to apply a more systemic lens to the same question. My conclusion is that by using some of the tools of antitrust economics we find a structural…
Postel’s Law famously states that “implementations should follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others.” For many years, this was considered a bedrock design principle for internet ecosystems, but in recent years it has fallen out of favor. In this post I will explain the deterioration that ecosystems which…
Canonical recently announced that they’re on the path to switching the coreutils that Ubuntu ships from being the venerable upstream GNU coreutils to a newer Rust implementation. This has made some people very excited and some people very upset (no one appears to be in the middle, though I suspect that’s just a sampling bias). This post will analyze some of the merits for/against this…
Toby: I read it, I think, 16 years ago. It was about El Salvador and he had it stricken from the record and there was a reason. Will: What? Toby: I don’t know, but things have reasons. Will: Do they? Toby: Yes, they do. – The West Wing, Inauguration Part I We might describe this as a Chesterton’s Fence sentiment. Chesterton’s Fence is the principle that one should not…
Once upon a time, a group of scientists built a machine learning model to predict whether a radiological scan contained a tumor. They trained it against a random subset of labeled data, tested it against data that had been held back, and determined that it performed well. When they went to test it with real patients, it was useless. Another group of researchers built a model to predict whether TCP…
In the United States, federal antitrust law is composed of several complementary statutes, one of the centerpieces of which is Section 2 of the Sherman Act.1 Section 2 prohibits monopolization (and attempted monopolization). Monopolization does not merely mean to posses a monopoly, rather it means (a) “possession of monopoly power in the relevant market”, and (b) “the willful…
For years, people have criticized the “SSO Tax”, SaaS companies' practices of limiting single-sign-on to only their more expensive plans, effectively imposing a tax on organizations that care about security. The critique is straight-forward: SSO is good for security, if you charge more for it fewer people will buy it, and you’ll get less security. Sometimes this is generalized…
Often in life, we want something from someone else. For example, as a citizen we want a politician to enact some policy, as a regulator we want a company to secure people’s data effectively, or as a customer we want a company to behave in a way we find ethical. A through line to these examples is that while we can clearly articulate the overall goal, just knowing the goal doesn’t…
I’ve had the good fortune to get paid to write open source as part of my job several times. For more than 15 years, I’ve also done a lot of open source development in my free time as a volunteer. Along the way, there’s been a fairly constant refrain that it’d be better if more open source maintainers were paid to maintain their projects. And I’ve seen a lot of ideas…
There are a variety of professions whose job it is to help businesses manage risks, for example, information security, accounting, and legal. And we generally expect those professions to reduce risk. But we also know that businesses take risks all the time, because while these professions reduce risk, and may even eliminate some risks, they don’t eliminate all risks. What is the…
I’ve maintained a popular cryptography library for years and in that time I’ve realized that many of the differences between cryptographic libraries flow from their creators' philosophies. Users of those libraries also have philosophies, which shape their expectations. These philosophies are often both in tension with one another, but also often implicit, leading to frustrating…
I’m indebted to a colleague who many years ago succinctly told me: Backups don’t matter, only restores matter. It’s a deceptively simple observation: if your goal is durability, what you truly care about is your ability to restore from a backup. The existence of the backup itself is meaningless – backups are not magic totems of durability. Teams that are serious about…
Sandboxing and memory safety are generally considered two orthogonal, and therefore complementary, approaches to improving security. Memory safety reduces the likelihood of a vulnerability being introduced, and sandboxing reduces the impact if a vulnerability is exploited. However, I think this over-simplifies what sandboxing looks like in sophisticated multi-process architectures like web…
I am an unrepentant advocate for migrating away from memory-unsafe languages (C and C++) to memory safe languages in security-relevant contexts. Many people reply that migrating large code bases to new languages is expensive, and we’d be better off making C++ safer. This is a reasonable response, after all there’s an enormous amount of C++ in the wild. Even on an incredibly aggressive…
I’m a staunch advocate for need to migrate away from memory unsafe programming languages, in order to address the endemic security issues they produce. This sentence contains a number of terms that are worth defining, and a number of asterisks that are worth explicating. My objective with this blog post is to add increased precision to this discussion. What is a memory unsafe language?…
One of my least favorite kinds of bug is when two different systems are interacting and the result has bad behavior but it’s difficult to say which (if either!) system is at fault. This is one of those stories, about Python’s buffer protocol and Rust’s memory model. Python buffer protocol Python’s Buffer Protocol is a set of APIs which allow Python objects to expose their…
Software packages have version numbers. Thinking about them from scratch, the first thing we might want from version numbers is to know if two pieces of software are the same, we could accomplish this by making version numbers into an opaque value, like a UUID. Of course, a UUID isn’t a very useful version number because in practice we also want to do things like order versions, to know…
Complexity in software, whether it’s a programming languages, an API, or a user interface, is generally regarded as a vice. And yet complexity is exceptionally common, even though no one ever sets out to build something complex. For people interested in building easy to use software, understanding the causes of complexity is critical. Fortunately, I believe there is a straightforward…
CSVs are a relatively popular data format, it seems particularly common as a format for providing exports of medium-sized datasets. My day job involves processing lots of these types of data sets, and so I’ve developed a set of strong opinions on CSVs, which are documented here. One feature of CSVs that is often considered a big advantage is that it’s an incredibly simple file format.…
There are not a lot of very strong empirical results in the field of programming languages. This is probably because there’s a huge amount of variables to control for, and most of the subjects available to researchers are CS undergraduates. However, I have recently found a result replicated across numerous codebases, which as far as I can tell makes it one of the most robust findings in the…
Every Passover, Jews around the world sing Dayenu, which translates roughly as “it would have been enough”. The lyrics are basically a list of things God did for the Jewish people, any of them alone would have been enough. “Taking us out of Egypt, that alone would have been enough. Giving us the Sabbath, that alone would have been enough. Giving us the Torah, that alone would…
software engineering is programming integrated over time This quote, from Titus Winters, expresses an important notion: that software engineering as a discipline must be considered not just with programming at a point in time, but with programming over an extended period. One of the things that tends to happen to a codebase as time is added, is that it tends to scale. We often think of scale in…
In January of this year, I put together a wish list for security in 2019. As the year draws to a close, I wanted to look back and reflect on what was accomplished, and where there’s still work to do. Rust breakthrough Original success criteria: Adoption of Rust as an official development language by another major OS and browser. Public talks/writing from teams that adopted Rust in these…
When you read “safety critical software” your mind probably went to something like medical device software (perhaps the Therac-25), or maybe avionics software, or perhaps industrial control systems that run nuclear power plants. It’s pretty unlikely that you thought of the operating system kernel on a cell phone or a web browser. Congratulations, you have a solid command of the…
I’ve been a ferocious critic of C, C++, and other memory unsafe languages, and a booster of memory safe languages such as Swift, Go, and particularly Rust. And though I believe there is a more-than-sufficient body of evidence to support the claim that the time to start migrating is now, there are still open questions related to how we migrate systems to memory safe languages more scalably,…
What is memory unsafety? Memory unsafety is a property of some programming languages where they allow the programmer to introduce certain types of bugs and allow these bugs to cause serious security issues. These bugs deal with errors in how memory is used spatially and temporally. To begin understanding these bugs, we’ll consider the example of an application that maintains to do lists for…
As software engineers, we overwhelmingly focus on the skill of writing code. This is clear from how we teach new programmers, how we interview software engineers for jobs, how we encourage engineers to improve their craft, and the kinds of talks you find at conferences. The lack of emphasis on developing code reading as a skill does people a disservice. Reading code is a distinct skill and…
I’m a frequent critic of memory unsafe languages, principally C and C++, and how they induce an exceptional number of security vulnerabilities. My conclusion, based on reviewing evidence from numerous large software projects using C and C++, is that we need to be migrating our industry to memory safe by default languages (such as Rust and Swift). One of the responses I frequently receive is…
It’s no secret that I’m a big believer in the adoption of security keys. I think they provide a strong technical solution to a problem that was previously unsuccessfully solved in user-hostile ways (don’t click on links in your email! look at the URL when entering your password!): security keys are the only second factor which are resilient to credential phishing. They’re…
Earlier this week the Google Security Team disclosed a pair of vulnerabilities, known to be exploited in the wild, one in Windows and the other in Chrome. These represent a fairly standard exploit chain: code execution in Chrome’s sandboxed renderer process and then a kernel bug to escape the sandbox and gain privileged code execution. There’s a publicly visible patch for the Chrome…
ImageMagick and GraphicsMagick are two popular libraries for manipulating images. GraphicsMagick is a fork of ImageMagick that diverged well over a decade ago. OSS-Fuzz provides continuous fuzzing for high impact open source projects. In December, 2017 Paul Kehrer and I worked to add ImageMagick to Google’s OSS-Fuzz, and in February, 2018 we added GraphicsMagick. Both ImageMagick and…
About 3 years ago I wrote about five projects I thought were very important for advancing the state of computer security. Looking back at that old post, I was reasonably pleased to find that all are having real positive impacts and none turned out to be busts. So I decided to take a stab at writing down the things I want to see happen in 2019, in the hopes that the universe will provide a few of…
When we write code, we optimize for many different things. We optimize for writability: how easy it is to write the code in the first place? We optimize for maintainability: how easy it is to make ongoing changes? We optimize for readability: how easy it is to understand what the code does? However, we rarely optimize for auditability: how easy it is to tell if the code has a security…
Attackers just need one vulnerability, defenders need to be perfect This may be the single most repeated truism in information security. Just this week, a colleague invoked this, with the quip that those of us who’ve chosen defense must be pretty dumb, given the challenge of that task, and the possibility of an easier career in offense. There’s just one problem: it’s not actually…
From 2015 to 2017 I worked for the United States Digital Service, a team within the US Government, created after the launch failure of healthcare.gov, dedicated to improving the government’s ability to use technology. I learned about a lot of different things there: bureaucracy and PowerBuilder, organizational transformation and Classic ASP, to name a few. However this post will instead be…
This past week Google’s Project Zero disclosed an unfixed security issue in Microsoft’s Edge browser. This is not the first time Microsoft failed to patch an issue within Project Zero’s disclosure timeline. This produces strong feelings in the information security community, generally in one of three categories: praising Google’s vulnerability research criticizing…
My day job is working on sandboxing for Firefox. In the context of a browser, sandboxing refers to the processes that run web pages, generally called “content” or “renderer” processes. These are in contrast to the “parent” or “browser” process, which coordinates the content processes and is not sandboxed, so it can do things like write files anywhere…
Heartbleed, POODLE, Shellshock. Giving vulnerabilities names may be controversial, but there’s no doubt it’s effective. These, and many other, vulnerabilities attracted widespread awareness and drove tons of work improving ecosystem security. Heartbleed drew attention to OpenSSL’s small team of maintainers and drove funding and code quality improvements. POODLE led to SSLv3 being…
This week Deputy Attorney General Rod Rosenstein gave two speeches on encryption; one at the U.S. Naval Academy and one at the Global Cyber Security Summit. I recommend you read them, as the remainder of this post will make considerably more sense. I would like to focus on the structure of the second speech. Mr. Rosenstein states that he wants to describe “the scope of the global…
If you’re a software engineer or work in tech, there’s a decent chance that your first thought after hearing about the Equifax breach was “oh my god, how incompetent do you have to be to get owned like that?” Don’t worry, I had the same reaction. After a few days of introspection and reviewing the evidence, I’ve come to the conclusion that Equifax made one…
There’s a lot of different types of work that tend to get put into the bucket “security engineering”. This goal of this post is to describe how I categorize different kinds of work, and why this is useful. At the highest level, security work goes into one of four buckets: Work that prevents us from getting owned. In this bucket are things like fixing bugs as well fixing root…
An alternate title for this post would be “Why GPG isn’t ok in 2017”. Imagine you were designing a new encrypted messaging system, what kinds of things would you worry about? You’d want to make sure you were using good encryption algorithms, authentication for senders, a high quality random number generator, maybe you’d spend some time thinking about side channels for…