Recent content on Jack's home on the web
The investigation timeline: From false DNS timeout leads to discovering the real culprit—a serial console bottleneck causing 3+ minute network outages. Abstract Link to heading Often for machine learning workloads, a single node of a large job crashing can stall the entire job. We run a very large AWS EKS cluster with large distributed workloads. This post is about investigating one issue where…
Abstract Link to heading We run a very large AWS EKS cluster with lots of interesting challenges. This post is about a recent investigation into DNS resolution failures that we were able to root cause to an Elastic Network Interface (ENI) packets per second (PPS) limit and a further root cause of the combination of sudo defaults and ndots in our cluster DNS. This post talks about the investigation…
Setup Link to heading Most cloud providers allow you to set commands to run when a host comes online. The general format for this is called cloud-init or user-data . There’s full documentation for cloud-init here . In my particular case, I had extra files I wanted to write to the host using pulumi . To make it all fit, I was gzip compressing the files then base64 encoding the result.
Fighting with hard drives TL;DR Link to heading Changing kubelet’s root directory can break CSI drivers and other components. You can work around this, but the effort isn’t worth it: stick with the default /var/lib/kubelet. Motivation Link to heading I was setting up Kubernetes kubelet on a host with numerous NVMe drives . I thought “Let’s use this for ephemeral-storage in…
Do you ever need to debug a node on kubernetes? Maybe the networking is broken, or you need to check something else out? The netshoot docker container contains a large number of helpful network tools like tcpdump, iperf, and more. Use a script like this to kubectl directly to a node with escalated privileges. Once in, try tcpdump -i any to see all network traffic on the node. #!/bin/bash set -exuo…
Image from UnSplash Preface Link to heading This post describes how I’ve been conducting sprint planning on my current team. It deviates from standard agile in a few ways. I document how and why towards the bottom. I’ve found this format works well for us, but there is no process that works for everyone. I script out the sprint planning process to keep it focused and make sure we…
Image from UnSplash I’ve led infrastructure at a startup for the past 4 years that has had to scale quickly. From the beginning I made some core decisions that the company has had to stick to, for better or worse, these past four years. This post will list some of the major decisions made and if I endorse them for your startup, or if I regret them and advise you to pick something else.
Context Link to heading We are using AWS Auora to manager our databases and I want to setup automatic rotation of database passwords. I start by storing them in AWS Secrets Manager . AWS actually releases publicly usable rotations for PostgreSQL , but I want to add the extra fields connstring and host_ro so clients can directly pull both the read only Auora DNS and a full PostgreSQL connection…
An unambiguous best practice for systems is to codify your infrastructure in a repeatable text format, rather than clicks in a UI. The common name for this is Infrastructure as Code . By far, the two most popular ways to do this, especially in AWS, are Terraform and CloudFormation . I used Terraform for about 3 years at a startup before working for Twitch (AKA Amazon Jr ) where I used Terraform…
Table driven tests are a popular model for testing in Go and a clear improvement to traditional testing of one function per case. However, there’s an alternative style that can often be clearer than table driven tests that relies on function closures . I’ll reference the example code on Dave Cheney’s post Prefer table driven tests for my examples. The original way Link to heading The code we want…
Histograms are an important part of metrics and monitoring. Unfortunately, it is computationally expensive to accurately calculate a histogram or quantile value. A workaround is to use an approximation and a great one to use is t-digest . Unfortunately again, there’s no such thing as an accurate approximation. There are various t-digest implementations in Go, but they do things slightly…
A recent vulnerability in the go-redis library allowed correctly configured clients to return the results of other redis commands in the event of redis failures. While all code is subject to bugs, this one could have been prevented if proper care was taken when creating and using interfaces. Bug behavior Link to heading Most clients reuse TCP connections between RPC calls to eliminate the delay of…
Problem Link to heading You have a constructor function that creates an object with a background goroutine. You want the object to garbage collect itself even if people don’t explicitly close it. This isn’t possible since since the background goroutine is always running and pointing to your created object. Solution Link to heading Wrap the returned object and use a finalizer on the returned object…
Problem Link to heading You’ve written an object or class. It’s somewhat complex and you want to iterate ways to better abstract your code but are unsure how to start. There exists documentation about what a good abstraction looks like, but no concrete iterative steps. Being deep inside your own code can make you blind to obvious alternatives. Solution Link to heading Create a bipartite graph of…
Problem Example Link to heading I commonly see bugs appending to slices in Go in a way that is not thread safe. A simple example is the unit tests below. This test has two goroutines append to the same slice. If you run this test with the -race flag, it works just fine. package main import ( 'sync' 'testing' ) func TestAppend ( t * testing . T ) { x := [] string { 'start' } wg := sync . WaitGroup…
I had an integration test that was very complex (most are, sadly) and took too long to start. I was able to diagnose why using new features of Go’s 1.10 trace tool. This post describes my workflow and what led me there. The example code I’m using for this post is at https://github.com/cep21/trace-110 . My integration test was a bit more complex, but generally had startup that was slow due to a…
A short checklist of what I wish for in a good Go library, in no particular order. This is a companion to the effective go list, Go code review comments list, and Go proverbs list. In general, when given two reasonable ways to do something, defer to the option that does not violate these rules. Bend these rules only with a strong explanation why. Dependencies Link to heading Tagged library…
What I wanted to do Link to heading Today’s pkg/errors Link to heading Package errors is a popular go error wrapping package. The StackTrace type represents the stack trace with an []Frame. Errors wrapped by package errors implicitly implement the stackTracer interface, which allows users to extract stack information. type StackTrace [] Frame type Frame uintptr type stackTracer interface {…
Problem Link to heading The go language spec has no way to simultaneously add or augment behavior to an interface while maintaining commonly accepted best programming practices around minimal interface design . While not unique to Go itself, a combination of Go’s feature set and evolved best practices have caused Go library authors to suffer disproportionately to other language developers.…
I get woken up to an alert about read capacity on one of my DynamoDB tables. Here is the error being logged. ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. When I load my CloudWatch graphs, and it appears I’m well below capacity. Cloudwatch graph of DynamoDB…
Go’s pprof package is frequently used for heap and CPU profiles, but a little used feature of the package is being able to create custom profiles. Custom profiles are useful for tracking down leaked resources and troublesome stack traces that are abusing a library’s API. Profile Link to heading From the godoc for pprof itself: A Profile is a collection of stack traces showing the call sequences…
Not lacking nuance Link to heading I’ve mentioned a general guideline of accept interfaces, return structs in a previous post and multiple times on code reviews to coworkers, but often get the question “Why”. Especially since this isn’t a hard rule. The crux of the idea, and understanding when to bend it, is in the balance of avoiding preemptive abstractions while maintaining flexibility. The…
I recently ran into a -race error message that stumped me for a while, so I’ll retell it here in phases to see if you can catch it. The code Link to heading The intent of the code was a service abstraction that ran a goroutine in the background to send runtime stats to Stastd . A small example of the code follows: type Stats struct { prev runtime . MemStats } func ( p * Stats ) stats () { ms :=…
An interesting property of Go’s built-in data structures is that read operations work even when they are nil! Not only that, but read operations on nil built-in structures behave just as if they are non-nil but empty. This is different from other languages and allows some interesting properties of structures in Go. Here is a full playground link . Why nil matters for built-in structures Link to…
Package httptrace is a new addition in Go 1.7 that allows users to trace the steps of a HTTP request. This package also models an interesting design pattern around adding optional debugging to a library. However, it’s not the only way to accomplish this goal. I’ll briefly overview the package and provide alternative implementations for httptrace. httptrace Link to heading The basic idea of…
This post will talk about a new library in Go 1.7, the context library, and when or how to correctly use it. Required reading to start is the introductory post that talks a bit about the library and generally how it is used. You can read documentation for the context library on tip.golang.org . How to integrate Context into your API Link to heading The most important thing to remember when…
This post describes what I feel are best practices when creating a client library for a service. The initial setup is that you’re writing a library with an API to talk to a RESTful service over HTTPS, and all the library needs to do is return JSON unmarshalled objects. I’ll use some examples from a client I wrote to talk to Smite’s API. Directly use the struct Link to heading While constructor…
This post will describe common approaches to organizing Go code into packages and what I feel is the ideal directory structure for Go repositories. Before starting with this post, it may be useful to also read https://blog.golang.org/package-names first for context. A setup with package proliferation Link to heading Let’s take an example application called gatekeeper. It may have a public client…
In this post I will describe the Preemptive Interface pattern often used in code and why I think it is generally an incorrect pattern to follow in Go. What is a Preemptive Interface Link to heading Interfaces are a way to describe behavior and exist in most typed languages. Preemptive interfaces are when a developer codes to an interface before an actual need arrises. An example may look like the…
I’m a software engineer and gamer. Here I post mostly technical stuff, but also some gaming related things. Find my posts on the blog page. Games Link to heading I play a lot of strategy games. Here are some of my favorites: Factorio Oxygen Not Included Slay the Spire Dyson Sphere Program Work history Link to heading Anthropic Link to heading At Anthropic I am part of the core-infrastructure…
I love to hear from people! For questions about specific posts, please leave a comment on the post. For general questions, send me a DM on LinkedIn or X . If you prefer email, use cep 221 at g mail d com.