I reckon I’ve drafted and then deleted a version of this post at least 10 times in the last 12 months. Deleted because it falls in the category “I must be wrong about this as everyone else is saying the opposite”. But this week’s release of Fable , and especially the reasons people are saying it’s such an improvement, are the nudge I needed to finally publish. So,…
Last summer I was about to start a new job where I’d be working with the Go programming language on a daily basis. Not having used it before and with a few weeks of freedom between jobs, I decided to implement a little chess engine in Go as a learning exercise. It turned out to be sufficiently interesting that I didn’t abandon the project and this week I reached a point where it…
This post is an expansion of a comment I made on HN recently and continues my Extreme Learning series. Previously I discussed how I’ve managed to break production with Redis , with PostgreSQL and with healthchecks . Now I’ll show you how I did it with Cloud Run too. For context, we migrated from Managed Instance Groups to Cloud Run at work last year. Cloud Run promises to simplify your…
When engineers discuss program design and system architecture, a common source of misunderstanding is concurrency. Often that’s because we make internal assumptions about it, which we presume to be self-evident. But we don’t all make the same assumptions, so you can end up in a situation where multiple conflicting beliefs are held about the concurrency of a system and nobody realises.…
Every technology stack has idiosyncrasies that can trip people up when they first encounter them. For whatever reason, I’ve particularly experienced this with engineers coming to Node.js from other platforms. Node is versatile, but there’s a specific paradigm of usage that can be surprising to the uninitiated. So, having been through this journey with people a few times now, here is my…
There’s a fascinating chapter in Keith Johnstone ’s Impro , called Status . The context is improvisational theatre but it contains some real wisdom about life in general. I’ve thought about it a lot over the years and it’s gradually informed some opinions I hold about healthy and unhealthy ways that engineering teams work together. The chapter opens: When I began teaching…
This post is an extension to a talk I gave recently at work. It was arranged at short notice and the audience were experienced engineers, so I kept it brief and tried not to be patronising. But the feedback afterwards was quite positive and there were some questions too, so here’s the extended version for anyone interested. Note that our team conducts pre-merge reviews, and some of the…
Recently I built a system that uses vector search to logically truncate long documents and retain the most significant parts according to some search term. I’m a dummy, with no background in machine learning or mathematics, so there were new concepts for me to understand and implementation details to figure out. This post summarises what I learned. What is vector search? Vector search is a…
For the last ten months or so I’ve worked on integrating GPT (various flavours) with a large production codebase. It’s been one of the most chaotic periods of my career, featuring numerous false starts, changes of direction and rapid improvements followed by major setbacks. This is what I’ve learned. AI assistants are only as good as their surrounding infrastructure Adding an LLM…
Production outages are great at teaching you how not to cause production outages. I’ve caused plenty and hope that by sharing them publicly, it might help some people bypass part one of the production outage learning syllabus. Previously I discussed ways I’ve broken prod with PostgreSQL and with healthchecks . Now I’ll show you how I’ve done it with Redis too. For the…
Previously for Extreme Learning , I discussed all the ways I’ve broken production using healthchecks . In this post I’ll do the same for PostgreSQL. The common thread linking most of these gotchas is scalability. They’re things that won’t affect you while your database is small. But if one day you want your database not to be small, it pays to think about them in advance.…
One reason I like working at startups is you get to wear many hats. Of course, by “wear many hats” I really mean “suffer occasional periods of extreme stress when things fail and there are no grownups you can go to for help”. I like to think of it as Extreme Learning . A hat I get to wear quite often is that of de facto devops consultant. And with that hat on, I’ve…
One of my stock interview questions goes: “When picking between dependencies to use in production, what factors contribute to your decision?” I’m surprised by how often I receive an answer along the lines of “Github stars” and not much else. I happen to think Github stars is a terrible metric for selecting production code, so this post sets out my idea of a healthier…
As engineers we spend a lot of our time debugging problems, yet it’s rarely taught as a skill in its own right. Some bugs are difficult enough that they can seem borderline impossible to solve, especially for devs toward the junior end of the spectrum. There’s no worse feeling than being stuck on a hard problem, not knowing how to proceed. Of course, the right thing to do if…
There’s nothing earth-shattering here, but I can recall that each of these principles were not obvious to me at some point in time. Perhaps some of them are not obvious to you right now. Always define explicit ON DELETE semantics If in doubt, use ON DELETE SET NULL Mutually exclusive columns Prohibit hidden nulls in jsonb columns Declare your updated_at columns NOT NULL to make sorting…
I’ve always wanted a dog, but could never get one because I lived in rented accomodation. Happily that changed last year so as soon as I could, I bought a puppy. This is what I learned about dogs, people and code. 1. Don’t be surprised when puppies do puppy stuff 🐶 As an engineer of somewhat advanced years, something I’ve struggled with at times is being patient with less…
Last week I received the result of my WSET level 3. I got a pass with distinction for both parts of the exam, the tasting and the theory. So, keeping up the tradition I started with level 2 , this post summarises my advice for anyone out there thinking about level 3. Theory papers As you’d expect, the size of the level 3 subject matter is a significant step up from level 2. There are more…
Automating your release process saves time, eliminates tedious busywork and reduces the likelihood of mistakes when cutting a new release. There are plenty of off-the-shelf solutions available, but this post will show how easy it is to build your own release script and why the end result can be better than using a generic, third-party option. Throughout the post I’ll use the case study of…
Historically, the Firefox Accounts (FxA) codebase was organised as separate repositories because it’s deployed as separate microservices in production. However, that arrangement caused us some nagging issues as developers, so we decided to migrate the code to a monorepo instead. This post discusses why and how we did that, and how things turned out in the end. Before There were three main…
Along with all the fun, creative stuff it enables you to do, programming sometimes requires you to carry out boring and repetitive editing operations. If those operations are uniformly applicable, it’s straightforward to automate them using regular expressions and a tool like sed , or :%s/foo/bar/g in vim-speak. But sometimes a regex can’t express the pattern you want to match against…
JavaScript’s Array and Promise types compose nicely with functional programming idioms to control concurrency without recourse to 3rd-party libraries. This post contrasts two such patterns that enable you to process data either concurrently or serially, with back-off and retry logic in the event of errors occurring. Concurrent execution Concurrent execution is the simple case, used when you…
Time passes and it’s easy to forget about all the cool stuff you did or lessons you learned. But source control makes it easy to go back and remind yourself, so I thought I’d take a moment to look back and summarise my past year in code. January January kicked things off with some improvements to our Redis abstraction and a number of small fixes to the Firefox Accounts metrics…
For Christmas I bought myself a Garmin Forerunner 645, after becoming irritated to the point of distraction by my Polar Flow M430. Now that I’ve used the Garmin for a few weeks, this is my compare-and-contrast review of the two devices. Polar Flow M430 These are the reasons I got annoyed with the M430: It takes a long time to connect to GPS and occasionally drops out part-way through a run,…
Recently I sat my WSET level 2 and did pretty well. I scored 90% in the exam, which is a pass with distinction. Based on that experience, these are my tips for anyone else planning to take the course. Before it begins, you will be sent copies of the textbook, workbook and a lexicon of terms to use during tastings. The workbook suggests you’ll need to do at least 28 hours of home study in…
Recently I wrote about how Rust macros make it easy to refactor repetitive code that might otherwise become annoying in a strongly-typed language. Continuing the theme from that post, I’ve noticed another use case where macros can be beneficial: writing custom assertions in tests. In the unicode-bom crate, there’s a bunch of tests that assert the different Unicode byte-order marks are…
About a month ago, we had an outage caused by a slow-running query in MySQL. This particular slow query wasn’t spotted when it was deployed because it depended on data inserted by client browsers and the related preference in Firefox was not enabled at that point. A few weeks after it shipped, the client pref was flipped on and as the table grew, it slowed down MySQL increasingly until the…
Refactoring boilerplate code is always easy in dynamically-typed languages, but sometimes takes a bit more effort when constrained by strong typing. This is something I was puzzling over recently, when the penny dropped for me about how Rust’s macros can be used to bridge the gap. If you have control over all of the code in question, macros probably aren’t needed of course. Some…
Richard Charles Maxwell Booth 25 th June 1948 - 17 th June 2018
Here’s three little Rust tricks that I learned in the last week or so. Each struck me as being an elegant approach to working cleanly within the confines of a strongly-typed language. Say you have a bunch of custom error types in different places, each tailored to the specific requirements of some module or function. When errors from lower levels bubble up through higher ones, they need to…
Let’s say you have a huge amount of JSON data and you want to parse values from it in Node.js. Perhaps it’s stored in a file on disk or, more trickily, it’s on a remote machine and you don’t want to download the entire thing just to get some data from it. And even if it is on the local file system, the thing is so huge that reading it in to memory and calling JSON.parse…
“Bad performance hurts user engagement” is a sentiment that feels intuitively true but can be hard to sell in product conversations. The best way to persuade a non-believer is to point them at hard evidence and for that you need to do a few things with your performance data. Firstly, don’t store it in a silo. That might sound obvious but silos occur incidentally for all kinds of…