Zero-downtime upgrade causes pipeline job retries to fail with missing ref error

## Summary Pipeline job retries fail with `fatal: couldn't find remote ref refs/pipelines/<ID>` error during Zero-downtime upgrades from 17.11.6 to 18.0.. ## Steps to reproduce 1. Set up a GitLab HA environment running version 17.11.6 2. [Perform zero-downtime upgrade](https://docs.gitlab.com/update/zero_downtime/) to 18.0.4 following the recommended order (Rails nodes first, then Sidekiq nodes) 3. After Rails nodes are upgraded but before Sidekiq nodes are upgraded: - Run a pipeline to completion - Retry a job from the completed pipeline ## What is the current bug behavior? Job retries fail with error: `fatal: couldn't find remote ref refs/pipelines/<ID>` ## What is the expected correct behavior? Job retries should work normally during zero-downtime upgrades, regardless of the order in which nodes are upgraded. ## Relevant logs and/or screenshots ``` fatal: couldn't find remote ref refs/pipelines/4740 ``` ## Possible fixes The issue stems from inconsistent behavior between Rails and Sidekiq nodes during the upgrade window: - Rails nodes (18.0.4) have feature flag `ci_only_one_persistent_ref_creation` enabled by default (FF was removed) - Sidekiq nodes (17.11.6) run with the feature flag disabled - When pipeline completes, ref cleanup in Sidekiq doesn't delete the `pipeline_persistent_ref_cache_key` Redis key because the cleanup code is gated behind the feature flag (https://gitlab.com/gitlab-org/gitlab/-/blob/v17.11.6-ee/app/services/ci/pipelines/clear_persistent_ref_service.rb#L7-L8) - On job retry, [CreatePersistentRefService](https://gitlab.com/gitlab-org/gitlab/-/blob/v18.0.4-ee/app/services/ci/pipelines/create_persistent_ref_service.rb#L17) runs in Rails with version 18.0.4, it finds the Redis key and skips ref creation, but the actual Git ref actually cleaned up. ## Environment - GitLab version: 17.11.6 → 18.0.4 - Deployment type: HA (Multi-node) - Upgrade method: Zero-downtime upgrade ## Additional context This issue was introduced by MR !182565 which added reliance on Redis cache for ref recreation. The feature flag `pipeline_persistent_ref_cache_key` was removed in 18.0 via MR !187651, creating the version mismatch scenario during zero-downtime upgrades.

issue