TL;DR: During my Harvester 1.6.1 → 1.7.0 upgrade, the etcd database hit 100% capacity and the cluster became stuck in a boot loop. Here’s how I recovered without data loss.
After initiating the upgrade to Harvester 1.7.0, my control plane node (whiterose) wouldn’t fully start. The rke2-server service was stuck in activating state, and the logs showed a troubling pattern:
level=info msg="Defragmenting etcd database"
level=info msg="Datastore using 93810688 of 93839360 bytes"
level=info msg="Failed to test etcd connection..."
That’s 93.8MB out of 93.8MB — etcd was at 99.97% capacity and stuck in an infinite defrag loop, unable to make progress.
etcd has a default storage quota (typically around 2-8GB, but Harvester uses a smaller footprint). During the upgrade, accumulated revision history and events filled the database to capacity. When etcd hits its quota, it goes into alarm mode and refuses writes — including the writes needed to complete defragmentation.
Harvester doesn’t put etcdctl in your PATH. It’s buried in the containerd snapshots:
find /var/lib/rancher/rke2 -name "etcdctl" -type f 2>/dev/null
I found it at:
/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12098/fs/usr/local/bin/etcdctl
ETCDCTL=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12098/fs/usr/local/bin/etcdctl
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
export ETCDCTL_CERT=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt
export ETCDCTL_KEY=/var/lib/rancher/rke2/server/tls/etcd/server-client.key
$ETCDCTL endpoint status --write-out=table
Output showed:
DB SIZE: 94 MB
IN USE: 94 MB
QUOTA: 0 B (display bug, but quota was effectively maxed)
First, get the current revision:
$ETCDCTL endpoint status --write-out=json | grep -o '"revision":[0-9]*'
# Output: "revision":132679682
Compact old revisions (use a revision slightly behind current):
$ETCDCTL compact 132679000
Then defrag to reclaim space:
$ETCDCTL defrag
After this, my status showed:
DB SIZE: 92 MB
IN USE: 32 MB
Percentage NOT IN USE: 66%
The compaction removed old revision history, and defrag reclaimed the physical space.
With etcd healthy, I tried starting rke2-server, but hit another issue:
Failed to test etcd connection: this server is a not a member of the etcd cluster.
Found [whiterose-97375ee3=https://10.0.0.144:2380],
expect: whiterose-97375ee3=https://10.0.0.145:2380
The node’s IP had changed via DHCP (from .144 to .145), but etcd had the old IP baked into its member configuration.
My UDM Pro had conflicting lease entries. I found them at:
cat /mnt/.rwfs/data/data/udapi-config/dnsmasq.lease | grep 85:32
Two entries existed for the same MAC — one for .144, one for .145. I removed the .145 entry and restarted dnsmasq.
On the Harvester node, NetworkManager was also caching the old lease:
cat /var/lib/NetworkManager/internal-*-mgmt-br.lease
# Showed ADDRESS=10.0.0.145
I deleted the cached lease and restarted NetworkManager:
rm -f /var/lib/NetworkManager/internal-*-mgmt-br.lease
systemctl restart NetworkManager
After a reboot, the node came up on .144, matching etcd’s expectations.
With the control plane healthy, the worker node (mrrobot) reconnected but showed the old version. The upgrade process had stalled, so I:
Deleted a stuck Longhorn PodDisruptionBudget that was blocking node drain:
kubectl delete pdb instance-manager-45f0a54a08a08991fd1306011e5653bf -n longhorn-system
Let the upgrade continue — it drained mrrobot, ran post-drain jobs, and rebooted the worker.
Both nodes came back healthy on Harvester 1.7.0:
NAME STATUS VERSION OS-IMAGE
mrrobot Ready v1.34.2+rke2r1 Harvester v1.7.0
whiterose Ready v1.34.2+rke2r1 Harvester v1.7.0
Know where etcdctl lives: In Harvester/RKE2, it’s in the containerd snapshots, not in PATH.
Compact before you’re full: Regular etcd maintenance (compaction + defrag) prevents this scenario. Consider adding a cron job or monitoring etcd size.
DHCP + Kubernetes = pain: Static IPs for cluster nodes avoid the IP mismatch nightmare. If you must use DHCP, ensure reservations are solid and leases don’t conflict.
Check both ends of DHCP: When fighting IP issues, check both the DHCP server (lease files, dnsmasq config) AND the client (NetworkManager cache, lease files).
PDBs can block upgrades: Longhorn’s instance-manager PDBs may need manual deletion if nodes are being drained during recovery.
Harvester keeps snapshots: I had etcd snapshots available in
/var/lib/rancher/rke2/server/db/snapshots/— valuable insurance if compaction hadn’t worked.
# Find etcdctl
find /var/lib/rancher/rke2 -name "etcdctl" -type f 2>/dev/null
# Check etcd status
$ETCDCTL endpoint status --write-out=table
# Get current revision
$ETCDCTL endpoint status --write-out=json | grep -o '"revision":[0-9]*'
# Compact (frees revision history)
$ETCDCTL compact <revision-number>
# Defrag (reclaims disk space)
$ETCDCTL defrag
# List etcd members
$ETCDCTL member list --write-out=table
# Update member peer URL (if IP changed)
$ETCDCTL member update <member-id> --peer-urls=https://<new-ip>:2380
# Check what's consuming etcd space
$ETCDCTL get / --prefix --keys-only | cut -d'/' -f1-3 | sort | uniq -c | sort -rn | head -20
Written after a late-night troubleshooting session. My cluster survived, the VMs came back, and I learned to not just revert to nuking everything.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.