## Summary Roll out [the fix](https://gitlab.com/gitlab-org/gitlab/-/issues/591291) currently behind the `retry_failed_keep_around_ref_writes` feature flag, introduced in [!248909](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/248909). `Gitlab::Git::KeepAround` tracked the Gitaly error and moved on when `write_ref` failed, so `MergeRequests::KeepAroundRefsWorker` reported success and its `retry: 20` never engaged, leaving the merge commit unprotected from `git gc`. The flag turns a failed write into a reported failure that the worker retries, and swaps the `commit_by` / `kept_around?` guard ordering so an unreachable Gitaly is recorded rather than silently skipped. - DRI: @marc_shaw - Team Slack channel: `#<slack-channel-of-dri-team>` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag is an actor-based project flag, and it changes behaviour in two places: `MergeRequests::KeepAroundRefsWorker` (which starts raising) and the inline `Repository#keep_around` callers `Ci::Pipeline`, `DiffPositionableNote` and `DraftNotes::PublishService` (which get the swapped guard ordering). The inline callers run in a web request — `Ci::Pipeline#keep_around_commits` is an `after_commit on: :create` — so a raise escaping `KeepAround#execute` would abort pipeline creation. Blast radius is every merge request save and every pipeline creation for an enabled project. **Watch while the flag is on, before removing it:** - `MergeRequests::KeepAroundRefsWorker` failure and retry rate — a small non-zero rate is the flag working; a sustained one means the retries are not converging. - Jobs landing in the Sidekiq dead set after 20 attempts. This is the permanent-failure case (corrupt ref file, repository genuinely gone) and should stay near zero. There is no `sidekiq_retries_exhausted` handler, and `KeepAroundRefsError` inherits `Gitlab::SidekiqMiddleware::RetryError`, so these are deliberately not in Sentry — the dead set is the signal. - Divergence between `gitlab_keeparound_refs_requested_total` and `gitlab_keeparound_refs_created_total`. The guard swap must not change either counter's meaning. - No new exceptions out of `Ci::Pipeline` creation or draft-note publishing. `Gitlab::Git::Repository::NoRepository` and `Gitlab::Git::CommandError` are rescued inside `KeepAround#execute`; `Gitlab::Git::ResourceExhaustedError` (Gitaly overload, circuit breaker open) is **not**, and escapes today on both flag states — confirm it stays at its current baseline rather than rising. - Contention on the keep-around write lease (`An identical job holds the keep-around write lease` in the worker logs). Expected to be rare; a rise means retries are overlapping with freshly enqueued jobs more than anticipated. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to the team channel. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set retry_failed_keep_around_ref_writes 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set retry_failed_keep_around_ref_writes true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set retry_failed_keep_around_ref_writes <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss retry_failed_keep_around_ref_writes true /chatops gitlab run feature set --group=gitlab-org,gitlab-com retry_failed_keep_around_ref_writes true /chatops gitlab run feature set --user=marc_shaw retry_failed_keep_around_ref_writes true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Removing the flag also removes the `check_ref_first` branch in `Gitlab::Git::KeepAround#execute`, leaving the ref check unconditionally first. ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete retry_failed_keep_around_ref_writes --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set retry_failed_keep_around_ref_writes false # production /chatops gitlab run feature set retry_failed_keep_around_ref_writes false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete retry_failed_keep_around_ref_writes --dev --pre --staging --staging-ref --production # remove entirely ```