Summer is over, took a break from the homelab and OSS projects. Came back to a large set of updates on the Proxmox hosts and a few PRs in calico-flow-logs-otlphttp-exporter (I’m starting to regret the name when having to spell it all out).

Proxmox

Since I don’t have a good solution for replicated storage for VMs I drain the K3s node VMs on each Proxmox host before rebooting after kernel updates. This saves time compared to migrating the worker node VMs and their 120GB storage drive across the network. It’s a compromise to minimize disruption without spending too much time on Proxmox updates.

Rebooting pve4 after having installed updates and drained nodes k8s-control-6, k8s-worker-1 and k8s-worker-4:

pve4 before reboot
pve4 down during reboot

Nodes were down as expected:

k get nodes -w
NAME            STATUS                        ROLES                       AGE    VERSION
k8s-control-4   Ready                         control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-control-5   Ready                         control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-control-6   NotReady,SchedulingDisabled   control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-worker-1    NotReady,SchedulingDisabled   <none>                      279d   v1.35.1+k3s1
k8s-worker-2    Ready                         <none>                      279d   v1.35.1+k3s1
k8s-worker-3    Ready                         <none>                      279d   v1.35.1+k3s1
k8s-worker-4    NotReady,SchedulingDisabled   <none>                      279d   v1.35.1+k3s1

pve4 after reboot:

pve4 after reboot

Uncordoning the nodes after VMs came back up:

k uncordon k8s-control-6
node/k8s-control-6 uncordoned
k uncordon k8s-worker-1
node/k8s-worker-1 uncordoned
k uncordon k8s-worker-4
node/k8s-worker-4 uncordoned
k get nodes -w
NAME            STATUS   ROLES                       AGE    VERSION
k8s-control-4   Ready    control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-control-5   Ready    control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-control-6   Ready    control-plane,etcd,master   279d   v1.35.1+k3s1
k8s-worker-1    Ready    <none>                      279d   v1.35.1+k3s1
k8s-worker-2    Ready    <none>                      279d   v1.35.1+k3s1
k8s-worker-3    Ready    <none>                      279d   v1.35.1+k3s1
k8s-worker-4    Ready    <none>                      279d   v1.35.1+k3s1
k8s-worker-2    Ready    <none>                      279d   v1.35.1+k3s1
k8s-worker-4    Ready    <none>                      279d   v1.35.1+k3s1

I usually do a small test to ensure VM migration isn’t broken (had this happen in the past after updates). A migration without any issues is a pass:

Testing VM migration after updates

Repeated for all Proxmox hosts.

calico-flow-logs-otlphttp-exporter

Released version 0.14.1 of calico-flow-logs-otlphttp-exporter. A few dependabot PRs failed to build, seems to be a breaking change in a new version of go.opentelemetry.io/otel/sdk/log. Ended up bumping all related dependencies in a new PR. Added a few quality of life changes to the the setup.

Updated: