Our Kubernetes platform test suite creates namespaces with their corresponding contents, then deletes everything during cleanup. We noticed a strange problem: namespace deletion would sometimes get stuck indefinitely. The root cause was surprising — we had to clear the contents before deleting the namespace! We also learned that getting stuck isn’t the only issue that can occur if we don’t do…
We ran into a weird pod scheduling error on Amazon Elastic Kubernetes Service (EKS). Some pods, which scheduled just fine in the past, now stay in Pending with the following event: Failed to schedule pod, incompatible with nodepool “default”, daemonset...
There’s a common Prometheus alert called “NodeMemoryMajorPagesFaults”. It’s part of the Node Exporter’s node-mixin project and also included in the kube-prometheus-stack default alert rules . But what does this alert mean, and what do you do about it? This article helps you form a good mental model and provides practical guidance.
Run a container with a host directory mount, and it either leaves root-owned files behind or it runs into “permission denied” errors. Welcome to the dreadful container host filesystem owner matching problem . These issues confuse and irritate people , and they happen because apps in the container run as a different user than the host user. There are various strategies to solve this issue , but…
I previously designed a robust distributed locking algorithm based on Google Cloud . Now I’m releasing a Ruby implementation of this algorithm: distributed-lock-google-cloud-storage-ruby . To use this, add to your Gemfile: gem 'distributed-lock-google-cloud-storage'