Lately, I've started to see more and more evidence that businesses are entering something of a "SaaS Hangover". In 2025 alone, 83% of companies reported experiencing a cloud data breach, and 70% of businesses using SaaS apps have lost data from those apps ( msp360 ). High-profile supply chain attacks have shown that even "safe" SaaS vendors can become attack vectors. Meanwhile, as geopolitical…
I've had the good fortune of building several large-scale software systems completely from scratch over the course of my career. While many of us spend our day jobs frustrated, making what seem like tiny changes in endless piles upon piles of code, starting from an empty directory and blank IDE can be a refreshing experience. In this environment, I find it to be especially rewarding when simple…
It always begins with a PagerDuty alert... A few weeks ago, I got a PagerDuty ping with this error: [FIRING:1] :fire: KubeContainerWaitingQuestDB Summary: Container in waiting state for longer than 1 hour Huh, well that could be caused by a bunch of things. Let's use kubectl to check the Pod's status: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image…
Originally published on the QuestDB Blog It was a little over a year and a half into my tenure as a cloud engineer at QuestDB when I started my first Pull Request to the core database. Before that, I had spent my time working with tools like Kubernetes and Docker to manage QuestDB deployments across multiple datacenters. I implemented production-grade observability solutions, wrote a Kubernetes…
The Kubernetes standard library is full of gems, hidden away in many of the various subpackages that are part of the ecosystem. One such example that I discovered recently k8s.io/client-go/tools/leaderelection , which can be used to add a leader election protocol to any application running inside a Kubernetes cluster. This article will discuss what leader election is, how it's implemented in this…
By default, operators built using Kubebuilder and controller-runtime process a single reconcile request at a time. This is a sensible setting, since it's easier for operator developers to reason about and debug the logic in their applications. It also constrains throughput from the controller to core Kubernetes resources like ectd and the API server. But what if your work queue starts backing up…
Check out Part 1 of this series for an introduction to Kubernetes operators. Now that we've covered the basics of what a Kubernetes operator is, we can start to dig into the details of each operator component. Remember that a Kubernetes operator consists of 3 parts: a Custom Resource Definition, a Custom Resource, and a Controller. Before we can focus on the Controller, which is where an…
If you work with persistent storage in Kubernetes, maybe you've seen articles about how to migrate from in-tree to CSI volumes , but aren't sure what all the fuss is about? Or perhaps you're trying to debug a stuck VolumeAttachment that won't unmount from a node, holding up your important StatefulSet rollout? A clear understanding of what the Container Storage Interface (or CSI for short) is and…
If you've used Kubernetes in just about any capacity over the past few years, you've almost certainly leaned on at least one operator to automate a piece of your infrastructure. The operator model has been espoused by influencers like Viktor Farcic (aka @DevOps Toolkit on YouTube ) and Kelsey Hightower for its advanced automation abilities. OperatorHub.io lists over 300 different operators for you…
I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting…
Kubernetes operators seem to be everywhere these days. There are hundreds of them available to install in your cluster. Operators manage everything from X.509 certificates to enterprise-y database deployments, service meshes, and monitoring stacks. But why add even more complexity to a system that many say is already convoluted? With an operator controlling cluster resources, if you have a bug,…
As you hopefully know from reading some of my recent articles, I really enjoy writing Kubernetes operators! I've written a bit about them on this blog , and also just recently presented a talk about how to build them at the inaugural FOSSY conference in Portland, Oregon. So here goes another operator-related article, although this one is about one of the later stages of operator development:…
This is a short article with a little k8s persistent storage nugget that I recently uncovered. FWIW, I've only checked this while using the AWS EBS Container Storage Interface (CSI) driver, but the article should apply to any cluster using CSI drivers to mount volumes. Background After a PersistentVolume (PV) is created and bound to a Persistent Volume Claim (PVC), a Pod can mount the claim as a…
It seems that annotations are everywhere in the Kubernetes (k8s) ecosystem. Ingress controllers, cloud providers, and operators of all kinds use the metadata stored in annotations to perform targeted actions inside of a cluster. So how can we leverage these when developing a new k8s operator? To the Docs Despite their widespread use, the official documentation of annotations is actually quite…
A few weeks ago, Kelsey Hightower wrote a tweet and held a live discussion on Twitter about whether it's a good idea or not to run a database on Kubernetes. This happened to be incredibly timely for me, since we at QuestDB are about to launch our own cloud database service (built on top of k8s)! You can run databases on Kubernetes because it's fundamentally the same as running a database on a VM.…
Recently, I've been spending a lot of time writing a Kubernetes operator using the go operator-sdk , which is built on top of the Kubebuilder framework. This is a list of a few tips and tricks that I've compiled over the past few months working with these frameworks. Log Formatting Kubebuilder, like much of the k8s ecosystem, utilizes zap for logging. Out of the box, the Kubebuilder zap…
Originally posted on the QuestDB Blog Introduction One of my favorite things about QuestDB is the ability to write queries in SQL against a high-performance time series database. Since I've been using SQL as my primary query language for basically my entire professional career, it feels natural for me to interact with data using SQL instead of other newer proprietary query languages. Combined with…
Thanks to a weekly Rust workshop hosted by one of my colleagues at QuestDB , along with some extra time spent in an airport lounge during a long layover, I was finally able to hack together some Rust that does something useful! questdb-retention The project I put together is questdb-retention , a small program that allows you to manage data retention in your own QuestDB instance. Even though…
My Fractal FM3 Setup I'm a longtime bass and guitar player who has lived in apartments for the better part of 15 years. Throughout this time, I've had a ton of "apartment friendly" guitar setups that used different combinations of effects pedals, amp simulators, headphones, low wattage amps, recording interfaces, and software plugins. Yet despite all of this tinkering, I've never been completely…
Originally posted on the QuestDB Blog Monitoring QuestDB in Kubernetes As any experienced infrastructure operator will tell you, monitoring and observability tools are critical for supporting production cloud services. Real-time analytics and logs help to detect anomalies and aid in debugging, ultimately improving the ability of a team to recover from (and even prevent) incidents. Since container…
Error handling in Go One of Rob Pike's famous Go Proverbs is that "Errors are values". This concept is deeply embedded in the go language and is seen in countless lines of source code; from multiple return values to if err != nil constructs. While treating errors like values provides you with error-handling flexibility that can't be found in most modern languages, sometimes you just need to check…
How to dynamically add nodes to a kind cluster Kind allows you to run a Kubernetes cluster inside Docker. This is incredibly useful for developing Helm charts, Operators, or even just testing out different k8s features in a safe way. I've recently been working on an operator (built using the operator-sdk ) that manages cluster node lifecycles. Kind allows you to spin up clusters with multiple…
Why nim? I first encountered nim at a PyGotham talk about optimizing hotpaths in Python code by actually rewriting the logic in nim and importing it back into your Python application. The language's Python-like syntax, static typing, easy-to-use FFI, and focus on performance appealed to me. Since that talk, nim was in the back of my mind as a new low-ish level language to pick up. Fast forward to…