I’m helping a recently acquired team at work figure out if they can migrate from Kafka to Google Cloud Pub/Sub. Part of the exploration was figuring out the change in latencies, if any, from switching. The team’s production setup is like this. They paid an external company called Confluent to run a managed Kafka cluster in AWS Oregon. This is the same region where this team ran all…
Last week I finished reading Michael Lewis The Premonition . The following parts of the book (with page numbers) stood out to me. The U.S. Centers for Disease Control (CDC) is portrayed as a risk-averse bureaucracy that wants to study disease and not take strong measures to control disease. Sometimes this interest conflicts with local health officials who want to save lives and see strong measures…
I collected mushrooms this morning with Gloria and my parents. It rained a lot last week in Waltham, MA. Many mushrooms had sprung up in Prospect Hill Park nearby. The temperature forecast for today said the highs would be 32°C. So I wanted to go out in the morning while it was not too hot. Over the objections of my mother, Gloria, my father, and I picked different mushrooms and carried them back…
I spent a long time figuring out how to install the latest grpcio Pip package (version 1.37.1) on my Apple M1 Macbook. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 pip install grpcio Looking in indexes: https://pypi.org/simple,…
This is a story about how debugged a confounding local development environment issue, what I learned about Docker in the process, and the generally applicable debugging strategies and techniques that helped me ultimately solve it. Skip to the end if you only want to read the debugging strategies and techniques. The overall story, however, will illustrate how they applied in this specific case.…
I recently observed DNS resolution errors on a large Kubernetes (K8s) cluster. This behavior was only happening on 0.1% of K8s nodes. But the fact that this behavior wasn’t self-healing and crippled tenant workloads in addition to my penchant to chase rabbits down holes meant I wasn’t going to let it go. I emerged learning how K8s Services Cluster IP feature actually works. Explaining…
I recently found more wargames at overthewire.org . Here are my hints and solutions for the first three levels of Vortex. The levels are cumulative. We have to beat the previous level in order to access the next. Vortex Level 0 -> Level 1 Hint 1: how much data Connect to the host and port and read all the bytes you can. How many bytes do you get? Hint 2: endianess “…read in 4 unsigned…
This is an explanation of Protostar level Final2 . I wrote a solution in April without an explanation. I read it last night and had to spend half a day to understand it again. So next time I’ll write the explanation while it’s still fresh in my head. The level’s description is Remote heap level :) Core files will be in /tmp. This level is at /opt/protostar/bin/final2 This is the…
Have you ever wanted to analyze the traffic between a mobile app and its servers or reverse engineer a mobile app’s non-public API? Here’s one way. The basic principle is to proxy the traffic from the app through a computer you control on which you can capture and analyze traffic. If the app you’re interested in is using an unencrypted protocol like HTTP, this is pretty easy.…
While stuck inside during social distancing, I’ve been making my way through LiveOverflow’s awesome Youtube playlist “ Binary Exploitation / Memory Corruption .” His videos are structured around a well known series of exploit exercises here called “Protostar.” I took the time to truly understand each one before moving onto the next as the exercises build on each…
In my previous post I wrote about how to load test GKE Workload Identity. In this post I’ll describe how to get metrics from gke-metadata-server, the part of Workload Identity that runs on your GKE clusters nodes. This solution is a temporary workaround until GKE provides a better way to get metrics on gke-metadata-server. Gke-metadata-server runs as a K8s DaemonSet. It exposes metrics about…
I manage multitenant Google Kubernetes Engine (GKE) clusters for stateless backend services at work. Google recently graduated GKE’s Workload Identity (WI) feature to generally available (GA). When my team used WI during its beta stage, it seemed to fail when there were more than 16 requests per second (RPS) on one GKE node to retrieve Google access tokens. Before we knew about this low RPS…
At the beginning of this year I set a goal of becoming a better public speaker and more visible in both tech and other broader causes I believe in. I’m happy to say that in the last two months I gave three talks! Two were prepared talks with slides at tech conferences. The other was an unprepared conversation on a podcast. These were all technical and related to my work at Spotify. Outside…
This is a post about Nginx’s DNS resolution behavior I didn’t know about but wish I did before I started using Kubernetes (K8s). Nginx caches statically configured domains once Symptoms I moved a backend service foo from running on a virtual machine to K8s. Foo’s clients include an Nginx instance running outside K8s configured with this upstream block. 1 2 3 4 5 6 7 8 9 10 11 12…
It’s been a while since I blogged about something tech related, but I had some free time today. Recently, I’ve been trying to refactor an internal Spotify deployment tool my team built and maintains. This deployment tool takes Kubernetes (k8s) YAML manifests, changes them, and essentially runs kubectl apply . We add metadata to the k8s manifests like labels. Right now this tool…
I’m compiling a list of Internet meme role models. Here’s what I have so far. These people — they must be real human beings — must have either gone out of their way to do the right thing in a smart manner, something courageous with bonus points for being funny, or just be ridiculous. And they must be memeified. Exceptions will be made for exceptional but not memeified…
I work at a music company but am more interested in politics and history. Artists visit our office often. So I, often being ignorant of their fame, have casually interacted with them or criticized their milk steaming techniques when they’re using the office’s $20K espresso machine. Only later am I told by their posse, “Did you know that was Mark Ronson/Bebe Rhexa/Max…
Whenever I go back home to my parents house near Boston, if my maternal grandparents are there, they make hundreds of dumplings for me. I try to help out. We make everything from scratch including the skins. I’m good at rolling the skins but have much to learn on all other parts of the process. I’m becoming better at packing and closing the dumplings now though. I’ve come to…
Here are the stories of four fascinating and weird people that will make you laugh, be inspired, or cringe. Chang and Eng Bunker were conjoined twins who married two sisters and were slave-owners on the side of the Southern Confederacy. Rose Wilder Lane is the daughter of the author who wrote the Little House childrens books, a founding member of the American Libertarian movement, and just all…
My home server’s hard disk’s partition map was somehow corrupted. So I’m serving this website from Digital Ocean for now instead of my apartment. While rewriting the nginx server configs, I found this useful site that tests your server’s TLS configuration . It’ll give you a grade and warn you of weak encryption, key exchange protocols, cipher suites, etc.…