Recent content on Jack Henschel's Blog
Recently I started seeing error messages like the following whenever tailing (following) Kubernetes pod logs in my k3s homelab: 1 failed to create fsnotify watcher: too many open files These errors are not just happening inside the Kubernetes environment, but also when I directly SSH onto the host and execute commands, such as: 1 2 3 4 5 6 $ journalctl -fu ssh Insufficient watch descriptors…
Recently, the machine which I’m using for selfhosting various services at home has been getting increasingly noisy. This was starting to be annoying since HP Prodesk G4 is located in my living room as this happens to be the most convenient spot because it’s right next to the Internet router. Especially given the fact that the fan inside this machine is tiny (it’s essentially a…
A common frustration among system administrators is user management: when I spin up a new tool in my cluster, the last thing I want to do is manage a separate set of credentials, or worse, hand out a single admin password for something like Grafana. I just want my users to log in with their existing cluster accounts. You can go the route of configuring Grafana’s generic OAuth2 integration ,…
I’m using Authentik as a single-sign on provider for all applications in my homelab. After a recent upgrade from Authentik to the latest version ( 2024.12.4 -> 2025.8.4 ), I was no longer able to log into my Gitea instance. After logging in via OIDC, Gitea simply greeted me with a 500 Internal Server error page. This is the moment every homelabber dreads: an upgrade breaks a critical service…
A month ago I had the pleasure of returning to Brno to speak at DevConf.CZ 2025 . This conference has a special place in my heart: it’s a fantastic community event where you can feel the passion for open source in every conversation. When my colleague Sven and I were assigned to a new customer project at the start of the year, we were quite skeptical about deploying the latest version of Red…
The cycling season is in full swing and I want to share some of my favorite tools for cycle route planning with you. Over the years, I’ve tested several online route planners, and I’ve settled on a few tools that I use for everything from short gravel loops to ambitious, multi-day mountain adventures. In this post, I’ll walk you through my go-to tools for planning cycling routes.…
RHOSO - the Red Hat OpenStack Services on OpenShift - is the brand new deployment model for OpenStack 18. In a nutshell, all the control plane components of OpenStack are containerized so they can easily and quickly be deployed in a Kubernetes environment. The data plane (compute hypervisors) remains separate on dedicated machines that are not part of the OpenShift cluster. For a recent customer…
In this post I want to share with you how you can use BATS - the Bash Automated Testing System - to create an end-to-end test suite for Kubernetes components. BATS can be used for many different purposes: testing command line tools, the behaviour of entire systems, and even APIs. To illustrate the capabilities of BATS, this post will show some practical examples for the kinds of tests that can be…
There are a number of reasons why it can be a good idea to install an alternative operating system (aka. Custom ROM ) on an Android device: better performance, more privacy, longer security updates and of course customization. Whatever the reason may be, users of custom ROMs (including myself) usually struggle with the fact that many banking apps and other “high security” apps refuse…
In this post I want to share a method I used recently for understanding the network activity of an application running on Kubernetes. Specifically the application I was looking at was velero (a backup and recovery tool): I could not figure out to which object storage endpoints it was talking to and which credentials it was using. I searched the internet for a transparent HTTP proxy that can…
As the birthplace of the world wide web , CERN has a long history of running web servers and web sites. More than 30 years later, we have built a scalable and multi-tenant platform based on the OKD Kubernetes distribution that allows us to handle an extremely diverse set of use cases. During my presentation at DevConf.CZ I discussed these different use cases and gave a technical overview of the…
Observability is an important and challenging part of any distributed system. Especially in the case of Kubernetes clusters this can be a tricky task due to the fact that the dozens of controllers (plus third-party operators) are loosely coupled, i.e. they all work independently. The kube-state-metrics component is a vital part of the Prometheus monitoring stack on Kubernetes: out-of-the-box it…
In this post I’d like to share some tools from my toolbox for working with container images (Docker/OCI). I specifically excluded tools for building container images (Buildkit, buildah, kaniko & friends). # What is a container image? Before we get to the tools, let’s first take a minute to understand what a container image actually is. This will allow us to better understand what these…
Last week all of our OpenShift (OKD) clusters started alerting us about the same degraded condition: 1 2 3 4 5 6 7 8 9 alertname = ClusterOperatorDown name = operator-lifecycle-manager-packageserver namespace = openshift-cluster-version openshift_io_alert_source = platform prometheus = openshift-monitoring/k8s reason = ClusterServiceVersionNotSucceeded severity = critical description = The…
I’ve been self-hosting Nextcloud (formerly OwnCloud) for almost a decade in my homelab. In this series I want to share the features and workflows that have proved to be most useful to me over the years. You can find all my posts about Nextcloud here . In the first part of this series I showed the most essential features of Nextcloud: file synchronization and sharing. In this post I’ll…
I’ve been self-hosting Nextcloud (formerly OwnCloud) for almost a decade in my homelab. In this series I want to share the features and workflows that have proved to be most useful to me over the years. You can find all my posts about Nextcloud here . Nextcloud is most commonly known as a personal file sharing and synchronization software, similar to Dropbox or Google Drive. Certainly for…
At the end of 2022, CERN’s web infrastructure suffered a major outage: thousands of internal and external websites became unavailable within minutes. My colleague Francisco Borges Aurindo Barros and myself were the ones doing the troubleshooting and cleanup on that day (and the following days). Almost a year later, we presented the incident at the USENIX Site Reliability Engineering…
Many web applications these days support some form of centralized user management (LDAP, OAuth2/OIDC, SAML …). For those that don’t a special type of reverse proxy can be installed “in front of” the application to handle user authentication. This authentication proxy makes sure all HTTP requests coming from outside are authenticated and only authenticated requests are…
Assume you have an idea for a new application or service. You write some Go to test and validate the idea - it works on your machine! What’s the simplest way to turn the source code into an easily shareable artifact? The answer is ko : ko makes building Go container images easy, fast, and secure by default. Some years ago I wrote about the ideal Dockerfile for a minimal Go container image -…
At the beginning of this year I gave a lecture and workshop at the Inverted CERN School of Computing 2023 . The CSC programme covers various aspects of scientific computing for high-energy physics and other data-intensive sciences. Naturally, my contribution was on the topic of cloud computing. During the lecture I covered the history and rise of cloud computing as well as some theoretical…
I’ve been self-hosting a Gitea instance (previously Gogs ) since roughly 2017. Since Gitea has historically been a very minimal Git hosting solution (note: nowadays it is very featureful!), it did not come with a builtin CI/CD solution (for testing, building, packaging, etc.). Thus I was also hosting a Drone instance to go along with it, which I’ve been using for my self-hosted,…
Kubernetes Events are ephemeral objects that indicate why and when a particular state changed in the cluster. They describe events such as when a pod was stopped (or failed), why a node become unhealthy, when liveness and readiness probes failed and many more (see also: A complete guide to Kubernetes Events ). 1 2 3 4 5 $ kubectl get events --sort-by = metadata.creationTimestamp AGE TYPE REASON…
Vector is a new, lightweight, resource-efficient and flexible log collector, aggregator and forwarder. I have written about Vector before and after using for a couple of different projects, I’m really fond of the characteristics of the tool, its documentation and ease of use. Ease of use also applies to Grafana Cloud: instead of having to set up Prometheus, Loki, Grafana et al. yourself (and…
This blog post walks through how to set up a log collector that ingests logs from an application, parses them and forwards them to an external logging system (such as Loki , OpenSearch or Kafka ) or archives on long-term storage (such as in a file or S3). The implementation section provides a step-by-step guide for parsing logs in the combined log format (used by NGINX and Apache) and sending them…
At the end of August 2023 I cycled along the Route des Grandes Alpes ( “route of the high alps” ) from Geneva to Nice. The route is quite famous for cyclist and motorbikes alike and has a dedicated website that describes several itineraries. All of the variants pass by iconic Tour de France and Giro d’Italia climbs such as Col du Galibier , Alpe d’Huez or Col de la Croix de…
I recently gave a lightning talk about Steampipe at CERN. The recording is available here , the following is a transcript of the talk. Cloud infrastructures tends to have lots of moving pieces: containers, loadbalancers, virtual machines, databases etc. Steampipe is a tool that allows querying all these pieces through a single interface with SQL. Are you tired of writing brittle Bash and JQ…
Looking around on the OperatorHub , the marketplace for Kubernetes Operators of various kinds, you will find many incarnations of “database operators”. However, most of them focus on either 1) running a database within Kubernetes (for example, the various Postgres operators) or 2) integrating with a cloud-managed database provider (such as Aiven). Unfortunately, none of them allow just…
Vector is the new kid on the block for collecting and aggregating logs. It’s similar to Fluentd , but one of the main benefits of Vector is that it is written in Rust, making it high performant and very efficient. Vector comes with its own domain-specific language ( DSL ) for transforming log records: the Vector Remap Language (VRL) . For one of the services we recently deployed, we want to…
Recently, we had an interesting case in our OKD cluster (which powers CERN’s internal Platform-as-a-Service) where one node after the other would report “DiskPressure” as well as “SystemMemoryExceedsReservation” conditions. The former means that the node is running on low on local disk space (used for storing container images, emptyDirs etc.), the latter refers to the…
OpenShift comes with powerful monitoring stack but in recent versions of OpenShift RedHat has progressively removed the ability to access the web interfaces of built-in Prometheus and Grafana instances (the OpenShift documentation refers to these as third party monitoring components ). Instead, the “Observe” view that comes with OpenShift’s Web Console should be used for this…
Container images are meant to be small and lightweight, since they contain only the required runtime dependecies of an application. Unfortunately, in the real world this often looks different: it is not uncommon to see container images that are hundreds of megabytes (or even gigabytes!) big. One of the issues that these large images cause are long delays when starting a container (or pod in the…
One of my recent posts included some footage from an onboard camera that was overlayed with a telemetry dashboard containing information such as speed, location, altitude, slope and a map. Since it took quite a while to find a software capable of producing this, in this post I’d like to share my workflow for creating these videos. I was specifically looking for a free and open-source…
After migrating my user accounts from OpenLDAP to Authentik , I still need to switch the Nextcloud authentication to a modern protocol, instead of relying on Authentik’s built-in LDAP outpost (which emulates an LDAP server). In a previous post a couple of months ago I explored how to connect the Nextcloud file-hosting suite to the Authentik identity provider with SAML. I managed to get the…
On May 22th, the Cyclotour du Léman 2022 was carried out. This is an annual, traditional cycling event around Lac Léman (Lake Geneva) happening in the Swiss cantons of Geneva and Vaud as well as the French department Haute-Savoie . This year, three routes were available: Lausanne to Evian (64 km), Evian to Lausanne (112km) and Lausanne to Lausanne (full lap around the lake, 176km). The starting…
In my homelab I’m self-hosting a couple of static websites from Minio S3 buckets - including the blog you are reading this article on. Using S3 buckets for static file hosting is great, because while the S3 interface was originally proprietary to AWS, it is nowadays widely supported by many tools and services. In addition, the S3 API comes with excellent built-in authorization primitives:…
The other day I was running some validation and performance tests for a new web service. To do this, I was effectively making a bunch of queries against a local HTTP endpoint. Both the server and client are written in Go, and the code querying the endpoint looks somewhat like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 func main () { //…
Since the beginning of this year, I have been looking for a new bicycle that I can use to properly explore the mountains and valleys of Switzerland and neighboring countries. Despite actively checking the second-hand market for quite a few months, nothing suitable turned up there - especially not for my size. Similarly, the supply chain situation on the new-bike market is still unbelievable: for…
On Sunday, May 15th 2022, the annual Journée Lausannoise du Vélo (JVL) took place. The ride starts at 7:30 am in a small village just outside the city of Lausanne , therefore I had to get up early to take the train from Geneva to Lausanne. Watching the sunrise over Lac Léman from the train The JVL is organized in two disciplines (mountain bikes and road bikes) as well various distances (ranging…
For years, I have been using using restic as my backup software of choice for servers and personal machines. There are several aspects I like about restic : it’s lightweight, fast and has a useful all-in-one CLI. On the technical side, it supports client-side encryption out-of-the-box and uses content-addressable storage to implement incremental backups and snapshots. Recently, I became…
While I was setting up my homelab with k3s, I was looking through the documentation of k3s and came across the --snapshotter argument. It allows changing the mechanism containerd uses for assembling the container image layers and isolating writes inside the container from the host (see containerd snapshot design ). Since my host system is using btrfs as its main filesystem, I thought it makes…
As part of my recent homelab overhaul, I was unsatisfied with my Drone CI/CD deployment. The main reason for this was the deployment needed to have a persistent volume, despite not storing any data - the volume was only needed for the SQLite database. SQLite is great for getting started, but proves to be inflexible when you have multiple machines and want to move your workloads around. Since I…
ArgoCD is commonly used to deploy and manage resources in Kubernetes clusters. One nice feature of ArgoCD is that it will continuously monitor the status of your resources (unlike for example Helm, which just creates the resources). For example, when your deployment fails to scale up to the desired number of replicas, ArgoCD will mark the Deployment as “Unhealthy”. ArgoCD has many…
In my previous post I described how to import user accounts from OpenLDAP into Authentik. Now I have my users in Authentik, so I want to connect Authentik with Nextcloud. There are several options available for this: 1: Run the Authentik LDAP Outpost and connect Nextcloud to Authentik’s (emulated) LDAP (Nextcloud has native LDAP support) 2: Use the Nextcloud “Social Login” app to…
For several years ( the earliest timestamp I could find is from 2016 ), I have been using OpenLDAP (a.k.a slapd ) as the source of truth for user accounts in my homelab. The initial setup was tricky – especially figuring out the “memberOf” overlay to get group memberships – but once I had it nailed down and put it into Puppet manifests , it was rock solid. The setup is…
While Kubernetes generally strives for backward-compatibility between minor releases, every now and then there are large changes in the Kubernetes API versions (with API versions I’m referring to the resources inside the cluster, not external language-bindings). For example, in Kubernetes 1.22 a large swath of v1beta APIs groups have been removed . For us as clusters administrators, this…
At work we have our own Grafana Kubernetes operator for deploying Grafana instances on OpenShift clusters. Since we are running the operator for our users, they cannot freely choose the version of Grafana themselves, but rather need to go through us. GrafanaLabs frequently releases new version of their software (awesome!) and we want to make those versions available as soon as possible to our…
While working on extending a Kubernetes operator built with the Operator SDK , I came across an unexplainable behavior in kubectl ( the Kubernetes CLI ) / oc ( the OpenShift CLI ), which I want to share in this post. The details of the operator are not relevant here, but my core goal was retrieving a list of all LoadBalancer services used in the cluster. Ideally, I would want to use something…
Kubernetes and OpenShift are highly complex solutions that allow you to work in the cloud. But sometimes you also need to come back down to earth again, for example when you have instability or data corruption issues with your kernel (see Fedora CoreOS Tracker #957 ). Then, you need to wade through layers of abstraction to understand how you can perform a simple task like changing the kernel of…
At work I came across a script that (was intended to) print out all “failed” Pods in a Kubernetes namespace. The script was executing the following command: 1 2 3 $ kubectl get pods \ --field-selector = 'status.phase!=Succeeded,status.phase!=Running' \ -o custom-columns = 'POD:metadata.name' Despite the fact that the command reads quite logically, it didn’t print out the expected…
The Archipelago Trail leads around dozens of the 40.000 islands in front of the Finnish city Turku . A friend of mine described this route as “green and blue heaven” and I think the term is pretty accurate: lots of lush forests, lots of sea water. In addition to cycling over the islands, the route also consists of several ferries used for hopping from island to island. This provides a…