diff options
| author | Kent Overstreet <kent.overstreet@linux.dev> | 2026-08-29 19:38:46 -0500 |
|---|---|---|
| committer | Kent Overstreet <kent.overstreet@linux.dev> | 2026-08-29 19:38:46 -0500 |
| commit | bc84205afe774c7bf475e66a9eb3b68144d62ebd (patch) | |
| tree | b8fc10af1ce51a8eb9980c6d421c82f2389f09ab /flake.nix | |
| parent | 0da3e5667804442c4113ec5b56545bc78d9d6912 (diff) | |
switch-to-configuration waits for a restarted unit's *start job* to
complete. This unit is Type=oneshot with TimeoutStartSec=infinity —
deliberately, so the timer can't kill a long distro rebuild — so that wait
is the length of an entire pass, hours on a backlog. It holds
/run/nixos/switch-to-configuration.lock for all of it, and every deploy
that lands in the window dies:
⭐ [activate] [DEBUG] Running activation script
Could not acquire lock
⭐ [activate] [WARN] De-activating due to error
⭐ [activate] [INFO] Attempting to re-activate the last generation
Could not acquire lock
Note the rollback fails for the same reason, which is how farm1 ended up
running a system that is in no surviving profile generation: the switch got
far enough to repoint /run/current-system before blocking.
farm1 wedged this way on 2026-08-29 20:41 — a switch sat 4 hours on a build
pass. Because the deploy hook does farm1 first and exits on the first
failure, farm2 got no config change for three weeks.
The unit is timer-driven and has no wantedBy, so restarting it on switch was
never useful: interrupting a pass mid-flight buys nothing that the next tick
doesn't give us for free.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Diffstat (limited to 'flake.nix')
| -rw-r--r-- | flake.nix | 12 |
1 files changed, 12 insertions, 0 deletions
@@ -468,6 +468,18 @@ # isn't in the system profile on hosts that only run podman via # this service (farm1: "default OCI runtime 'crun' not found"). path = [ config.nix.package pkgs.podman pkgs.crun pkgs.openssh (builtins.dirOf config.security.wrapperDir) ]; + # NEVER restart this on a config switch. switch-to-configuration + # waits for a restarted unit's start job to *complete*, and for a + # Type=oneshot with TimeoutStartSec=infinity (below) that means the + # whole pass — hours on a backlog. It holds + # /run/nixos/switch-to-configuration.lock the entire time, so every + # later deploy dies with "Could not acquire lock", including the + # rollback. farm1 wedged this way on 2026-08-29: a switch sat 4h on + # a build pass, and because the deploy hook does farm1 first and + # exits on failure, farm2 went undeployed for three weeks. + # It's a timer-driven batch job, so restarting is pointless anyway — + # interrupting a pass buys nothing the next tick doesn't give us. + restartIfChanged = false; serviceConfig = { Type = "oneshot"; User = cfg.user; |