Recent content on Roberto's blog
Some decisions require comparing multiple options where it’s not immediately apparent which is best. One common way of doing that is to list the pros and cons of each option. For example, option A lists low latency as a pro, while option B lists high throughput as a pro. But do you know what throughput option A provides? Not being explicit makes it easy to make assumptions and ignore crucial…
I regularly use different devices with different OSs, such as a MacBook, a Windows desktop with WSL, a couple of Raspberry PIs, and so on. I have a bunch of tools I like to have at my fingertips when I log into a machine, like fd and Neovim. Setting up and maintaining the same configuration across all of my machines using different package managers was painful enough that I just reverted using…
I have been using Gatsby for my blog and I have never been fond of it’s complexity considering that all I need is Markdown rendering. Trying to update Gatsby and its dependencies to the latest version was painful enough to motivate me to jump ship. After looking at what’s out there, I settled for Hugo. As I will inevitably forget all about it in a month’s time, I am writing this…
Coordination is expensive as it reduces the availability and performance of distributed applications (PACELC theorem). I have extended chapter 10 of Understanding Distributed Systems with a discussion of how to minimize coordination using one of the following patterns: Keep coordination off the critical path Use protocols that guarantee some form of consistency without coordination Proceed without…
I am excited to announce that the first edition of my book about distributed systems is finally complete! First things first, I have rebranded the book from “The Distributed Systems Manual” to “Understanding Distributed Systems,” as I felt it was more appropriate for the content. It’s the second time I change the title, but also the last, I promise :) I have also…
Sometimes a single process in a system needs to have special powers, like being the only one that can access a shared resource or assign work to others. To grant a process these powers, the system needs to elect a leader among a set of candidate processes, which remains in charge until it crashes or becomes otherwise unavailable. When that happens, the remaining processes detect that the leader is…
At scale, any failure that can happen will eventually happen. Hardware failures, software crashes, memory leaks - you name it. The more components you have, the more failures you will experience. This nasty behavior is caused by cruel math - given an operation that has a certain probability of failing, as the total number of operations performed increases, so does the total number of failures. In…
I have released a new chapter of Understanding Distributed Systems! It explores the different patterns at your disposal when designing horizontally scalable applications. The simplest way to scale an application is by running it on more expensive hardware. But, that only brings you so far as the application will eventually reach a performance ceiling. The alternative to scaling up is scaling out…
An application typically starts its life as a monolith. Take a modern backend of a single-page Javascript application, for example - it starts out as a single stateless web service that exposes a RESTful HTTP API and uses a relational database as a backing store. The service is composed of a number of components, or libraries, that implement different business capabilities: As the number of…
I have released a new chapter of Understanding Distributed Systems: Resiliency Patterns. The chapter is all about failures and their mitigations. Any failure that can happen will eventually happen at scale - hardware faults, software crashes, memory leaks - you name it. The more your system scales out, and the more failures it will experience. Eventually, the only way to cope with them is with…
The system design interview is a great way to assess the seniority of a candidate in an interview. You can find a lot on the Internet on how to prepare for the design interview or which system design interview questions to expect, but very little on how to conduct an actual interview. In this post, I will describe my experience interviewing senior candidates for distributed systems roles. If you…
Imagine assigning some value to a variable, reading it back immediately after, and finding out that somehow the write had no effect at all - madness! x = 42 assert(x == 42) # throws exception This is precisely what can happen when you are using a distributed data store with weak consistency guarantees. “But wait, aren’t databases supposed to take care of consistency issues for…
In the second chapter of Understanding Distributed Systems, I explore the core building blocks at the heart of many distributed systems. What can you expect to learn from it as a reader? In a distributed system, anything that can fail will eventually do so. But, you can only mitigate a failure if you can detect it in the first place. Hence, the second chapter starts by introducing the concept of…
I have released the first chapter of Understanding Distributed Systems! “Wait, what? Weren’t you working on a video class?” - I hear you ask. As I was recording the video lectures for the class, I realized I had to write a lot before recording a lecture - recording is the easy part, framing what to say in a way that is easy to understand and correct is the hard part. The more…
The response times of your service can cost you dearly if left unchecked. Even when a small fraction of requests experiences extreme latencies, it tends to affect your most profitable users, and not in a good way. On top of that, long response times can decrease the resilience of your service, making it more costly to operate. But, I am getting ahead of myself - to begin with, let’s start by…
There are two types of engineers, the ones that can quickly do estimates and the ones that can’t. Are these people just smarter, or is there more to it? Enrico Fermi was a master at estimation Back of the envelope estimation is a skill that can be learned with practice. It becomes easier once you get familiar with some tricks of the trade. Know your numbers How fast can you read data from…
Why do you need to place your servers geographically close to your users? One of the reasons is to achieve lower latencies. That makes a lot of sense when you are sending short bursts of data that should be delivered as quickly as possible. But what about large files, such as videos? Surely there is a latency penalty for receiving the first byte, but shouldn’t it be smooth sailing after…
Modern applications don’t crash; they hang. One of the main reasons for it is the assumption that the network is reliable. It isn’t. When you make a network call without setting a timeout, you are telling your code that you are 100% confident that the call is going to succeed. Would you really take that bet? If you are a making synchronous network call that never returns, then to very…
I get asked this question a lot. I lost track of people I know that spend all their time brushing up on algorithmic puzzles and barely prepare for the system design round. I can’t blame them; it’s easy to find comfort reversing lists, finding shortest paths, and whatnot. The solutions of those problems can be checked for correctness or efficiency. But you can’t automatically…
It’s been three years since my last post on my old Wordpress blog! You can still find my earlier posts there as I was too lazy to move them all over here. Plus, most of those posts are no longer relevant anyway these days. A lot has changed in the past three years. In July 2017, I moved to Microsoft to work on an internal data platform as a SaaS product.
If a user has opted into submitting performance data to Mozilla, the Telemetry system will collect various measures of Firefox performance, hardware, usage and customizations and submit it to Mozilla. Telemetry histograms are the preferred way to track numeric measurements such as timings. The histogram below is taken from Firefox’s about:telemetry page. It shows a histogram used for…
Technology allows companies to collect more data and with more detail about their users than ever before. Sometimes that data is sold to third parties, other times it’s used to improve products and services. In order to protect users’ privacy, anonymization techniques can be used to strip away any piece of personally identifiable data and let analysts access only what’s strictly…
Writing good code is hard, writing a good analysis is harder. Peer-review is an essential tool to fight repetitive errors, omissions and more generally divulge knowledge. I found the use of a checklist to be invaluable to help me remember the most important things I should watch out for during a review. It’s far too easy to focus on few details and ignore others which might be catched (or…
How engaged are users for a certain segment of the population? How many users are actively using a new feature? One way to answer that question is to compute the engagement ratio (ER) for that segment, which is defined as daily active users (DAU) over monthly active users (MAU), i.e. $$ ER_{segment} = \frac{DAU_{segment}}{MAU_{segment}} $$ Intuitively the closer the ratio is to 1, the higher the…
This is a short post on the elegance of using abstract algebra for analytics in Scala. A monoid is a set \( T \) that is closed under an associative binary operation \( append \) with an identity element \( zero \) such that \( append(a, zero) = a \). In other words, the following 3 properties apply: Closure - the result of combining two elements of the set is also an elment of the set: \[ \forall…
Spark execution model Spark’s simplicity makes it all too easy to ignore its execution model, and still manage to write jobs that eventually complete. With larger datasets, having an understanding of what happens under the hood becomes critical to reduce run-time and avoid out of memory errors. Let’s start by taking our good old word-count friend as starting example: rdd =…