Debugging writeups, infrastructure notes, references, code, and technical articles for people who work close to systems.
With modern distributed systems performance issues can surface in unexpected ways. When it comes to database servers like PostgreSQL issues can surface as slow pages, API calls timing out, jobs not completing, or users saying that the application is "slow". These issues usually occur at 4AM, so having a good debugging methodology (and lots of coffee) baked into your brain can help with speedy…
Prometheus is a monitoring system that collects time series metrics from targets at regular intervals. Those metrics are stored with labels, queried with PromQL (Prometheus's custom query language), and graphed in tools like Grafana. In a typical setup, Prometheus scrapes metrics from exporters running beside the systems you care about: a node exporter that exports system metrics like CPU, memory…
Software supply chain attacks target the dependencies, build systems, and publishing pipelines that produce the software we run, instead of the application itself. A single compromised package can reach millions of systems before anyone notices, and once it lands, it runs with the same trust and privileges as the rest of your code. The diagram below shows where these attacks typically land along…
Over the past few months I've been doing diving deep into PostgreSQL. I've spent a good bit of my career supporting Oracle and MySQL, but over the past few years I've spent more time managing PostgreSQL. I'm super comfortable getting around a psql prompt, but wanted to really dive into the guts of PostgreSQL to take my knowledge to the next level. I started with the PostgreSQL internals…
It has been a while since I last posted in 2022. A lot has changed since then: the pandemic reshaped how we work, software and infrastructure practices continue to evolve, and AI has become one of the most significant technology shifts I have ever seen. I have always loved learning, and AI has accelerated that process in a meaningful way. It makes it easier to ask focused questions, understand…
Over the past several years I've read numerous horror stories about cloud deployments gone wrong. S3 buckets with PCI data left open to the raw Internet, EC2 instance profiles that weren't scoped properly, misconfigured NSGs, etc. It takes a LOT of time to truly understand all the ins and outs of running workloads in the cloud, and making sure you get it "right". This is one reason I'm always on…
Having worked in the "cloud" for several years, one thing that I'm super conscious about is our cloud bill. There are tons of subtleties associated with billing, such as AZ-to-AZ traffic costs or how VPC endpoints can reduce egress charges. If you utilize Terraform for infrastructure provisioning, you may want to look at infracost. Infracost can help you understand cloud spend for a green field…
The growth of the Terraform community is absolutely astounding. New providers are constantly popping up, providers are being upgraded at a feverish pace, and amazing new features are constantly being added. With all of this change, deprecations and breaking changes periodically surface. One way to protect yourself from breaking changes is to pin providers and modules to specific versions…
When I was first getting started with Kubernetes, RBAC was one of the topics that took me the longest to grok. Not because the resources (Roles, ClusterRoles, etc) are hard to interpret, but learning how to scope your Roles to minimize access takes some practice. That and a lot of reading to understand the various API groups and what they contain. In a previous post I mentioned access-matrix,…
Debugging production issues can sometimes be a challenge in Kubernetes environments. One specific challenge is debugging containers that don't contain a shell. You may have seen the following when troubleshooting an issue: Not including a shell in your base image is a best practice, and projects like distroless make it super easy to package your applications with a small shell-less footprint. But…
Last month I started a course that teaches you how to write your own Operating System. Working at the intersection of hardware and software (X86 Assembly and C) has been incredibly rewarding. I've learned a TON! One interesting thing I came across in the Linux kernel's bootloader code is the use of "asm volatile"…
Container images are one of the items that makes up a "container." In most cases container images use a base image (e.g., Alpine, Ubuntu, etc.), and then one or more application-specific layers are added on top of that. There are numerous documented best practices for optimizing container images, and these best practices result in smaller images, less network traffic, and a reduction in container…
One of my friends recently reached out with a fun problem. His monitoring system was periodically not firing when file systems grew past the thresholds he defined. When we hopped on one of his EC2 instances to debug the issue, I noticed that we were getting a permission denied (EACCES) errno when running df as their monitoring user: When we ran the same command as trusty UID 0, everything worked…
If you've worked with Kubernetes for any length of time, you are probably intimately familiar with deployment manifests. If this concept is new to you, deployment manifests are used to add resources to a cluster in a declarative manor. Some of the larger projects (cert-manager, Istio, CNI plug-ins, etc.) in the Kubernetes ecosystem provide manifests to deploy the resources that make their…
This past week I got to spend some time upgrading my CI/CD systems. The Gitlab upgrade process requires stepping to a specific version when you upgrade major versions, which can be a problem if the latest version isn't supported by the upgrade scripts . In these types of situations, you can tell yum to upgrade to a specific version. To list the versions of a package that are available, you can use…
Kubernetes has truly revolutioned distributed computing. While it solves a number of super hard problems, it also adds a number of new challenges. One of these challenges is ensuring your Kubernetes clusters are designed with failure domains in mind. Designing around failure domains includes things like provisioning infrastructure across availability zones, ensuring your physical servers are in…
Ansible has amazing support for testing services during playbook execution. This is super useful for validating your services are working after a set of changes take place, and when combined with serial you can stop execution if a change negatively impacts one one or more servers in your fleet. Ansible has a number of modules that can be used to test services, including the uri module. The uri…
As a Kubernetes administrator I frequently find myself needing to debug application and system issues. Most of the issues I encounter can be solved with Grafana dashboards and Prometheus metrics, or by running one or more Elasticsearch queries to examine logs. But there are times when I need to go deeper and actually inspect activity inside a running pod. A lot of debugging guides use the kubectl…
This week I was updating some Ansible application and OS update playbooks. By default, when you run ansible-playbook it will apply your desired configuration to hosts in the order they are listed in the inventory file (or in the order they are returned by a dynamic inventory script). But what if you want to process hosts in a random order? Or by their sorted or reverse sorted names…
I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So I started to decode all of the HTTP requests and responses from a $(docker pull), which truly helped me solidify how the docker daemon (dockerd) talks to a container registry. I figured I would share…