Our Kubernetes cluster's DNS recently experienced partial degradation. We noticed it via DNS resolution errors in Sentry mostly when calling internal services. This incident took a bit to root cause - not that it was particularly thorny in retrospect, per se, but it involved diving into the guts of things one doesn't normally care much about. While this isn't the only interesting incident that…
Recently we've needed to do a lot of HTTP requests to weird APIs. One endpoint in particular is basically the wild west of HTTP - High, unpredictable latency and redirects a lot: often absolutely, from HTTPS to HTTP and even to non-resolving URLs. I started this endeavour with good ol' HTTP.rb , as one does, blissfully unaware of this long tail of responses. I configured a 15s timeout for this…
We've hit a couple of bugs during a recent feature launch. This feature's 1.0 is an email - so no takesies backsies, you better get it right the first time. Unfortunately, this was not the case (curse you string types . Maybe don't store booleans in CSVs? Maybe don't use CSVs as persistence. But I digress). We initially shrugged off one of them during the first batch of emails, but it came back…
I'm using highlight.js for syntax highlighting in this blog, client-side. I was looking to pre-compute it statically, so I wrote a little node.js helper that reads stdin, highlights the html using the library, and prints to stdout (not only is it more unixy, I already have too many temp files being created). So to call it from ruby, I found the shell gem . 1 It used to be in the stdlib and the DSL…
Ok hi! One of my first projects ever was a blog written in python flask (please don't look at it, it was 2015). Even though I shut it down a couple of months after having it up on my server (a VPS at the time), my admiration for blogging didn't stop and, of course, I read a bazillion blog posts since then. But now, here we are, finally! My own tech blog, v2.0. Just some flat hand-made HTML files…