apenwarr - NITLog
Happy July 4th! For those of us around the world contemplating independence, it's a good day to think about how we came to rely on expensive cloud infrastructure for our fundamental computing needs. With that in mind, here is my latest toy project: an open source tool that makes replicating, forking, sharing, and running container snapshots fast and easy across cloud and personal devices. It's fun…
In recent months I've heard of several teams with an interesting policy: each pull request should be no more than a few files, and no more than a certain number of lines (say 500). And do just one thing and do it well. And be easy for a human to review. And be fully tested by the test suite. All those are good requirements, right? Surely this is quality software engineering. And often, the results…
AI isn’t killing engineering. It's making it less meditative. There’s a lot of talk about AI killing engineering jobs. Some jobs will change, and some will disappear. But we’ve been automating engineering work for decades, and somehow we keep finding more engineering to do. At Web Summit 2026, I talked about how AI can raise the floor for developers, and what that means for juniors trying to break…
AI is the cause of, and solution to, slop problems. Have AIs filter your PRs so humans get back to deciding what they want to exist. AI was built on open source. Now it’s starting to return the favour: finding bugs, writing fixes, and optimizing code faster than humans can review the diffs. At Web Summit, I sat down with the CEO of Cal.com to talk about what happens when AI agents and open-source…
AI needs an open ‘Android’ to balance the vertically locked-in ‘iPhone’ of trillion-dollar AI companies. Open-source AI is closing the gap with closed systems faster than many people expected. That changes the economics, the security model, and the assumptions behind a lot of Big Tech’s AI spending. I joined a press conference at Web Summit to talk about what happens next: who benefits, which…
AI is the universal translator that finally connects all your tools... into a hairball. Most companies don’t have a tool shortage. They have too many tools, too many handoffs, and too many places where context gets lost. I joined a panel at Web Summit to talk about when consolidation actually reduces work, where AI helps, and how to avoid replacing ten mediocre tools with one very large mediocre…
We’ve all heard of those network effect laws: the value of a network goes up with the square of the number of members. Or the cost of communication goes up with the square of the number of members, or maybe it was n log n, or something like that, depending how you arrange the members. Anyway doubling a team doesn't double its speed; there’s coordination overhead. Exactly how much overhead depends…
Long ago in the 1990s when I was in high school, my chemistry+physics teacher pulled me aside. "Avery, you know how the Internet works, right? I have a question." I now know the correct response to that was, "Does anyone really know how the Internet works?" But as a naive young high schooler I did not have that level of self-awareness. (Decades later, as a CEO, that's my answer to almost…
I have a friend who exited his startup a few years ago and is now rich. How rich is unclear. One day, we were discussing ways to expedite the delivery of his superyacht and I suggested paying extra. His response, as to so many of my suggestions, was, “Avery, I’m not that rich.” Everyone has their limit. I, too, am not that rich. I have shares in a startup that has not exited, and they seem to be…
Our company recently announced a fundraise. We were grateful for all the community support, but the Internet also raised a few of its collective eyebrows, wondering whether this meant the dreaded “enshittification” was coming next. That word describes a very real pattern we’ve all seen before: products start great, grow fast, and then slowly become worse as the people running them trade user love…
Treat AI agents like a naive, high-energy intern. As AI takes off, it’s bringing new security challenges—especially around how models are built and accessed. In this talk, I sat with Cohere Co-founder Ivan Zhang dig into the often-overlooked networking layer behind AI and why secure, reliable connections are becoming essential for enterprise-ready systems. If you'd rather read than watch, the full…
The Internet’s been broken since IPv6 stalled; Tailscale fixes it at the IP layer. Before I knew exactly what Tailscale would become, I knew how I wanted to build it: keep the product simple, sell to developers, win small markets before expanding, and avoid inventing management structures for fun. I talked with Amit Kumar about those early decisions, what worked, what didn’t, and how Tailscale…
An executive with 8,000 indirect reports and 2000 hours of work in a year can afford to spend, at most, 15 minutes per year per person in their reporting hierarchy... even if they work on nothing else. That job seems impossible. How can anyone make any important decision in a company that large? They will always be the least informed person in the room, no matter what the topic. If you know me,…
The Net Promoter Score (NPS) is a statistically questionable way to turn a set of 10-point ratings into a single number you can compare with other NPSes. That's not the good part. Humans To understand the good parts, first we have to start with humans. Humans have emotions, and those emotions are what they mostly use when asked to rate things on a 10-point scale. Almost exactly twenty years ago, I…
A few conversations last week made me realize I use the word “interesting” in an unusual way. I rely heavily on mental models. Of course, everyone relies on mental models. But I do it intentionally and I push it extra hard. What I mean by that is, when I’m making predictions about what will happen next, I mostly don’t look around me and make a judgement based on my immediate surroundings. Instead,…
I really like the "tech debt" metaphor. A lot of people don't, but I think that's because they either don't extend the metaphor far enough, or because they don't properly understand financial debt. So let's talk about debt! Consumer debt vs capital investment Back in school my professor, Canadian economics superhero Larry Smith , explained debt this way (paraphrased): debt is stupid if it's for…
Someone asked if I could write about the rise of AI and Large Language Models (LLMs) and what I think that means for the future of people, technology, society, and so on. Although that's a fun topic, it left me with two problems: I know approximately nothing about AI, and predicting the future is hard even for people who know what they're talking about. Let's try something else instead. I'll tell…
I just did a presentation at SREcon Conversations (which I call SREconcon) EMEA, called the "Curse of Unreasonably Sized Networks." I talked about the series of Dunbar's numbers and how they relate to different kinds of human social networks, and surprisingly, also to the evolution of the Internet. Video recording of the talk (34 minutes) Slides (pdf) SREcon EMEA 2020 speaker's page
Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also needed to be updated in a hurry, but won't be until they facilitate some ransomware, which is pretty normal for these situations.) I have a 20+ year…
What if all these weird tech trends actually add up to something? Last time, we explored why various bits of trendy technology are, in my opinion, simply never going to be able to achieve their goals. But we ended on a hopeful(?) note: maybe that doesn't matter. Maybe the fact that people really, really, really want it, is enough. Since writing that, I've been thinking about it more. I think we…
The Internet has gotten too big. Growing up, I, like many computery people of my generation, was an idealist. I believed that better, faster communication would be an unmitigated improvement to society. "World peace through better communication," I said to an older co-worker, once, as the millenium was coming to an end. "If people could just understand each others' points of view, there would be…
Let me tell you about the still-not-defunct real-time log processing pipeline we built at my now-defunct last job. It handled logs from a large number of embedded devices that our ISP operated on behalf of residential customers. (I wrote and presented previously about some of the cool wifi diagnostics that were possible with this data set.) Lately, I've had a surprisingly large number of…
I guess I know something about train wrecks. One night when I was 10 years old, me and my mom were driving home. We came to a train crossing outside of town. There was another car stopped right on the tracks, stalled. A lady was inside, trying to get her car to start. It didn’t. Train crossings are bumpy, cars were worse then, it was a long time ago, I don’t know, I don’t remember clearly. Anyway,…
Let's talk about bug/feature tradeoffs. Anyone who knows me has probably already heard me rant about Crossing the Chasm , my most favourite business book of all time. I love its simple explanation of market segmentation and why the life cycle of a tech startup so often goes the way it does. Reading that book is what taught me that business success is not just a result of luck or hard work.…
Welcome to this week's edition of "building a startup in 2020," in which all your meetings are suddenly remote, and you probably weren't prepared for it. I know I wasn't. We started a "fully remote" company back in 2019, but that was supposed to mean we still got together in person every month or two to do strategic planning, share meals, and resolve any accumulated conflicts. Well, not this year.…
As a "fully remote work" company, we had to make some choices about the technologies we use to work together and stay in touch. We decided early on - about the time we realized all three cofounders live in different cities - that we were going to go all-in on remote work, at least for engineering, which for now is almost all our work. As several people have pointed out before, fully remote is…
"Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines. You might think that's not a possibility. Back in university, students used to tease the Systems Design Engineers, calling it "boxes and arrows" engineering. Not real engineering, you see, since it didn't touch anything tangible, like buildings, motors, hydrochloric acid,…
No.
A few years ago I wrote The World in Which IPv6 was a Good Design . I'm still proud of that article, but I thought I should update it a bit. No, I'm not switching sides. IPv6 is just as far away from universal adoption, or being a "good design" for our world, as it was three years ago. But since then I co-founded a company that turned out to be accidentally based on the principles I outlined in…
Long ago, I wrote git-subtree to work around some of my annoyances with git submodules. I've learned a lot since then, and the development ecosystem has improved a lot (shell scripts are no longer the best way to manipulate git repos? Whoa!). Thus, I bring you: git-subtrac . It's a bit like git-subtree, except it uses real git submodules. The difference from plain submodules is that, like…
I made a little flow chart of mainstream programming languages and how programmers seem to move from one to another. There's a more common kind of chart, which shows how the languages themselves evolved. I didn't want to show the point of view of language inventors, but rather language users, and see what came out. It looks similar, but not quite the same. If you started out in language A, this…
I was recently recommended to read the book Accidental Empires by Robert X. Cringely , first published in 1992 (or was it 1991?) and apparently no longer in print and also not in e-book format. To my surprise, it turns out archive.org has a solution for this, an epically user-unfriendly "virtual library card" (which is still worth it if you need to read a book) in which they apparently receive one…
Multi-factor authentication remains hard-to-use, hard-to-secure, and error-prone. I've been studying authentication lately to see if it might be possible to adapt some security practices, especially phishing prevention, from big companies to small companies and consumers. Here's what I have so far. (Sorry: this is another long one. Security is complicated!) The three factors aren't independent The…
I don't mind letting your programs see my private data as long as I get something useful in exchange. But that's not what happens. A former co-worker told me once: "Everyone loves collecting data, but nobody loves analyzing it later." This claim is almost shocking, but people who have been involved in data collection and analysis have all seen it. It starts with a brilliant idea: we'll collect…
Back in the early 2000s, XML was all the rage. An unusual evolution from HTML, which itself was an evolution (devolution?) from SGML, XML was supposed to be a backlash against complexity. SGML originally grew from the publishing industry (for example, the original DocBook was an SGML language ) and had flexible parser features so not-too-technical writers could use it without really understanding…
I've been working on a series of tutorials using redo for various use cases. (One of the most common user requests is more examples of how to solve real-world problems with redo. The problem with extremely flexible tools is that it can be hard for people to figure out how to start.) The most recent in the series is a a tutorial on building docker and kvm containers from scratch using redo . I…
From a book I've been reading: Power is certainly important, particularly in dictatorships, in places where constitutions, laws, unwritten rules, traditions and understandings don't count. But in a healthy democracy, power is a surprisingly limited element. And the unwritten conventions, understandings, forms of respect for how things are done, for how citizens relate to government and to each…
A while ago I was discussing hiring plans with a co-founder of a recently-funded startup. Their story was something like this: "The company is growing very fast and we now have about a dozen people. We have tight deadlines, and none of the founders has much experience in project management or people management. So, we're trying to hire an experienced project manager and an experienced engineering…
[ Note 2017-12-29: the news.ycombinator.com discussion of this post is unusually useful. You may want to read it first.] [ Note 2018-09-01: I presented an updated and shortened version of this talk at SREcon EMEA 2018. There's a recording and some slides that you might enjoy. The text below is much more detailed though, if you want to actually implement the advice.] I did a talk at work about the…
I recently decided to switch my laptop from a Macbook to a Chromebook, partly because Apple's keyboards are so terrible lately, and partly because ChromeOS is suddenly useful now that they invented Crostini . (Some people ask why I, a person who actually knows how to use Linux and has debugged wifi drivers and XF86Config files, would want to use a locked-down desktop Linux variant instead of just…
tl;dr: Rebuilding a target because its mtime is older than the mtimes of its dependencies, like make does, is very error prone. redo does it better, and so can you. A recent twitter discussion (pro tip: never do those) led me to realize that I have studied the problem of mtime comparison in considerably more depth than most people. I don't know whether to be proud of this or very concerned, but…
I'm working on (statistically speaking) my least interesting work in years. Previously I spent time doing wifi drivers, boot scripts, logfile processors, payment systems, and project estimation, so one might reasonably have assumed I can't get much more boring. But hah! We have standards to exceed. Here's what's weird though: I'm enjoying it. There's this Feeling I get, very rarely, when I'm sure…
Lately I've been getting back to hacking on my djb redo implementation . We've fixed some problems with file attribute handling on NFS, obscure locking, and MacOS/FreeBSD portability. If you haven't tried redo in a while, you might want to give it another shot. In case you haven't heard of redo before, here's the overview: it's like make , but with no special syntax (just sh scripts). The first…
Here's a post I've been thinking about for a while. I shared it privately with a few people last year, but wanted to find a way to present it that wouldn't be wildly misconstrued. No luck so far. Summary: Inspired by some conversations at work, I made a variant of my previously popular SimSWE (software engineer simulator) that has our wily engineers trying to buy houses and commute to work. The…
My parents live in a rural area, where the usual monopolist Internet service provider provides the usual monopolist Internet service: DSL, really far from the exchange point, very very asymmetric, and with insanely oversized buffers (ie. bufferbloat), especially in the upstream direction. The result is that, basically, if you tried to browse the web while uploading anything, it pretty much didn't…
I updated my SWE simulator[1] from a few weeks ago. This time, instead of PMs changing their minds about features, we have PMs and execs who choose not to make up their minds at all, letting launch requirements get bloated. These four charts have the same eng team, same set of features, same pattern of bugs getting filed, same work rate, etc. The only difference is how many "features" (one "bug"…
I spend a lot of time answering various people's tech-business questions. Occasionally, I will say something that sounds brilliant, and an unlucky listener will exclaim, "Avery, how do you know so much stuff?" Answer: I read it in a book. Now, I don't actually read very many books. But I've been lucky enough to receive some great recommendations. Here are mine. Now you, too, can sound like a…
One of the dangers of working at a big company that does a little bit of everything, even if that big company has a pretty open culture, is it's really hard to remember which things are still secret and which have been released. Since I started working for such a company back in 2011, I've still had as many opinions as ever, but I've contained them mostly to an audience at work, so that I don't…
, you will need to go through a montage of embarrassing and destructive mistakes as you learn to use those superpowers.
keep posting about. But my deep dark secret is... I mostly got it from reading books in the first place.