Efficient query execution in distributed data warehouses depends on how well the workload is balanced across nodes. Napa improves performance by dynamically partitioning data at query time, adapting to each query’s needs instead of relying on fixed partitions. It uses a progressive approach designed to be “good enough” thereby balancing partitioning time and performance. A typical data warehouse…
Maybe it’s because I’m an immigrant desperately trying to fit in. Maybe it’s one more way of keeping the imposter syndrome at bay. Whatever the reason, I’ve always been low-key obsessed with software development lingo. I’d go through meetings dropping principles and laws every chance I got. I’d lurk Slack channels waiting for a chance to pounce on a conversation…
When I first joined the Netflix Platform team circa 2020, the Observability offering was composed of a series of tools serving different purposes. There was Atlas for metrics, Edgar for distributed tracing, Radar for Logs and Alerts , Lumen for dashboards, Telltale for app health, etc. It was a portfolio of about 20 different apps. Big and small, ranging from business-specific tools to analyze…
This is the door to my in-laws’ apartment. You can safely ignore the puppy; it’s only there for SEO purposes. Focus instead on that thing on the wall, marked by the orange arrow. That is a motion-sensor light switch, the kind that turns the light on when you walk by. Only this one doesn’t because it works like shit. You must wave your hand in front of it, almost touching it, for the lights to come…
Pete Hodgson explored the different ways in which a Platform team works with other teams to get shit done in this article . I thought it was interesting to see how collaboration changes based on the type of work, so I put together this visual summary to compare and contrast each type of interaction. Click image to enlarge I added a few things here and there, but most of the stuff is taken from the…
Pivot Tracing lets users define arbitrary metrics over trace data at runtime. It does so by combining two powerful techniques: A Happen-Before operator that allows users to perform queries based on the causal relationship of the events. The ability to instrument code dinamically without having to redeploy. How it works Say we have an instrumented system we want to troubleshoot: The first step is…
Extension functions are great! But if you define them all over the place, it can get confusing pretty quickly. So here’s a cool idiom to limit extension function usage to a specific context. Last week I needed to write some code to generate Atlas Stack Language (ASL) queries. ASL is loosely based on Reverse Polish Notation , so you first specify the parameters and then the operation. The query I…
Distributed tracing can be ridiculously expensive if you try to trace a hundred percent of requests. A common technique to reduce costs is to sample only a small portion of the traffic. But naive sampling techniques like uniform sampling will inevitably capture more common-case executions and might miss the more interesting edge cases. Instead, Sifter’s approach is to bias sampling decisions…
Lots of businesses run on Google Docs. It’s how we write memos, define strategies, discuss proposals, document decisions, write tutorials, and plenty of other things. Google Docs is a fantastic piece of technology. I almost can’t imagine how we worked before it ( productStrategy-Jun-2004-version13.docx anyone?). And yet, I sometimes feel like it could be so much more! Like we’ll…
Common knowledge says that you don’t deploy on Friday if you want to have a peaceful weekend. Yet, some people will tell you that if you’re not comfortable deploying every day of the week, you’re doing it wrong. They’ll say that deploying shouldn’t be scary and that you probably don’t have enough tests. So, which one is it? The problem with Fridays Fridays are the last day of the working week, and…
It’s Monday morning. You’re sitting at your desk with your steaming cup of Joe, ready to sink your teeth into that new feature you have to develop. The git pull downloads months worth of changes, and you dive into the code. Piece by piece, you start building a mental model of the system, trying to make sense of the different components. But something doesn’t feel right. Why was it built this way?…
What we include in a test is as important as what we leave out. Having the right amount of information helps us understand what the test is doing at a glance. Let’s say we need to check our Restaurants are behaving correctly. We want to validate two things: That a restaurants can only cook a Recipe if they have all the necessary ingredients. That vegan restaurants do not serve non-vegan food…
Flaky tests are those that randomly fail for no apparent reason. If you have a flaky test, you might re-run it, over and over, until it succeeds. If you have a couple of them, the chances of all passing at the same time are slim, so maybe you ignore the failures. You know, just this one time… Soon enough, you’re not paying attention to failures on this test suite. Congratulations! Your tests are…
Using expressive assertions can help us figure out why a test fails without having to go through the code. Let’s start with an example. Here’s a test making sure our recipe has tomatoes 🍅 At first glance, everything looks ok. The test passes, it is easy to read, and it follows the ”Given - When - Then” structure . Some months go by, and one day our test starts failing. At this point,…
One way to make sure your tests are readable is to have them all adhere to the same structure. By far, the most common structure is “Given - When - Then” (aka “Arrange, Act, Assert” ). It goes like this: On Given : We create the objects and set up the needed state. On When : We perform the action we want to test. On Then : We validate the state changed as expected. For example: The comments…
Picking good test names can help us identify what’s wrong with our code when something fails. It’s Friday afternoon. You finally finished that long refactor you’ve been working on for the whole week. Everything is looking good. Except you run the tests and see one failure. 🤔 Unfortunately, you can’t really tell what’s broken from looking at that output . You’ll have to…
Queues are a powerful tool for building reliable systems. In this article, I’ll describe some of the tips and tricks I came across when working with queues. Some of the advice is specific to Amazon SQS queues because that’s what I’ve been using the most lately. And also because some of them come from this amazing article from the Amazon Builders’ Library . The trade-off Queues can be used to…
We have deluded ourselves into thinking that being able to invert a binary tree on a whiteboard is the hallmark of great software engineering. It’s time we look for better ways of evaluating coding skills. Hello, my name is David. I would fail to write bubble sort on a whiteboard. I look code up on the internet all the time. I don't do riddles. — DHH (@dhh) February 21, 2017 Why whiteboard…
This is a compilation of all the things I learned for creating effective presentations. The golden rule: NO BULLET POINTS If you can only take one thing from this article let it be this: DO NOT use bullet-points on your slides . There’s a simple reason for this: Humans brains are really dumb. They can’t process spoken and written words at the same time. People will either read the slides or…
In this article, I’ll show how to implement a Spring Boot application using Hexagonal Architecture. We’ll build a Bank Account simulation with deposit and withdraw operations exposed through REST endpoints. Hexagonal Architecture Hexagonal architecture is an architectural style that focuses on keeping the business logic decoupled from external concerns . The business core interacts…