harvester upgrade stuck at “post draining”

I recently kicked off an upgrade on my Harvester cluster from v1.7.0 to v1.7.1. Everything was looking good — the upgrade image downloaded, the repo was created, nodes were prepped. Then it just stopped. The UI showed “Upgrading Node 0%” and one of my nodes was stuck at “Post-draining” with no signs of life.

After some digging, I found the post-drain pod was looping the same message over and over:

Upgrade repo deployment is not ready yet, waiting...

Not exactly helpful on its own. So I started pulling threads.

When Harvester runs an upgrade, it spins up a deployment called upgrade-repo that serves the upgrade ISO over Nginx. By default, it wants 2 replicas of this deployment. Makes sense for redundancy. But here’s where it falls apart in a small cluster.

I’ve got two nodes — mrrobot and whiterose. The upgrade drains whiterose first and marks it SchedulingDisabled. Now both upgrade-repo pods have to land on mrrobot. The first pod comes up fine. The second pod? Stuck in ContainerCreating.

The reason: the Longhorn volume backing the upgrade ISO only had a replica on whiterose — the node that just got drained. Longhorn stopped that replica when the node was drained, so it can’t attach the volume for the second pod. You end up with this error:

volume pvc-xxxxx hasn't been attached yet

The deployment is sitting at 1/2 available, so Available=False. The post-drain script checks that condition in a loop every 10 seconds. It never passes. The upgrade is deadlocked.

SSH into your cluster and run through these:

# Check the post-drain pod logs
kubectl -n harvester-system logs -l harvesterhci.io/upgradeComponent=postDrain --tail=20
# Check the upgrade repo deployment — look for AVAILABLE vs READY
kubectl -n harvester-system get deployment -l harvesterhci.io/upgradeComponent=repo
# Look at the pods — one will be stuck in ContainerCreating
kubectl -n harvester-system get pods -l harvesterhci.io/upgradeComponent=repo -o wide
# Check where the Longhorn volume replicas live
kubectl -n longhorn-system get replicas -o custom-columns=\
NAME:.metadata.name,VOLUME:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState

If you see the deployment at 1/2 available and a pod stuck trying to mount the volume, that’s your problem.

This part is straightforward. The first pod is already running and serving the ISO just fine. We don’t need the second replica to complete the upgrade. Scale it down:

# Find your deployment name
kubectl -n harvester-system get deployment -l harvesterhci.io/upgradeComponent=repo
# Scale to 1
kubectl -n harvester-system scale deployment <upgrade-repo-deployment-name> --replicas=1
# Verify
kubectl -n harvester-system get deployment <upgrade-repo-deployment-name> \
  -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'

You should see True. Within about 10 seconds the post-drain script picks it up and the upgrade starts moving again.

In my case, the scale-down got things moving initially, but then the upgrade hit a second wall. The Longhorn volume backing the upgrade-repo went completely unresponsive. The Nginx container’s readiness probe started failing, and the post-drain script couldn’t download rootfs.squashfs — it was pulling a 0-byte file.

The root cause was the Longhorn instance manager. When whiterose got drained and marked SchedulingDisabled, the instance manager pod on that node couldn’t reschedule. Without a running instance manager, Longhorn couldn’t start the volume replica. The volume went from attached to detached, and Nginx had nothing to serve.

Check for this with:

# Look at instance manager status
kubectl -n longhorn-system get instancemanagers -o wide
# Check the volume state
kubectl -n longhorn-system get volumes -o custom-columns=\
NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,NODE:.spec.nodeID

If you see an instance manager in error state with no pod, and your volume is detached — that’s the problem.

The fix? Uncordon the drained node:

kubectl uncordon <node-name>

This lets Kubernetes schedule the instance manager pod back onto that node. Once it’s running, Longhorn can start the replica, attach the volume, and Nginx comes back to life. The post-drain script will pick up right where it left off.

I know it feels wrong to uncordon a node mid-upgrade — the whole point is that it was drained for a reason. But in a two-node cluster, Longhorn needs at least one healthy instance manager per node to keep volumes attached. The upgrade process doesn’t account for this, and the node will get drained again when it’s actually time to apply the OS update.

After the uncordon, the volume reattached, the rootfs downloaded successfully (978MB), the OS update applied, and whiterose rebooted into v1.7.1. The whole thing took longer than it should have, but it got there.

This is going to hit you if you’re running a two-node Harvester cluster or any setup where the Longhorn replica for the upgrade PVC only lives on the node being drained. If you’re planning an upgrade, keep an eye on two things:

1. The upgrade-repo deployment — if it’s stuck at 1/2 available, scale it to 1.

2. Longhorn instance managers — if volumes start detaching after the drain, check whether the instance manager can run. Uncordoning the node temporarily will get things moving.

If the post-drain stage stalls for more than a couple minutes, odds are it’s one of these two issues.

Cheers,

Joe