## Summary Operational rollout tracking for the `log_labkit_rack_divergence` ops flag introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/245389. The flag gates a temporary sampled diagnostic log for the labkit rack shadow: one `application_json.log` line per divergent block decision between labkit and Rack::Attack, carrying both stacks' verdicts, counters, and discriminators. It exists to root-cause the parity divergence classes tracked in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362, which block the per-cohort enforce gates. - DRI: @mwoolf - Team Slack channel: `#g_production_engineering` > [!note] > This is not a feature release. The flag is a sampling dial for a diagnostic log line: it will be raised for bounded diagnosis windows, returned to off between them, and removed together with the log once https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362 is diagnosed. "Stable at 100%" is explicitly not the goal. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) ## What could go wrong? - **Log volume.** Only divergent decisions are logged (never matches), currently a few hundred per second fleet-wide at worst, dominated by `throttle_authenticated_git_lfs` during active LFS throttling events. 100% sampling during such an event is the worst case; prefer raising the percentage outside LFS abuse events or keeping windows short. Watch the divergence-by-throttle dashboard (linked from the work item) to predict line rate before raising. - **No user-facing behavior.** The logging happens on the outbound path after the response is decided, inside the middleware's guard: a logging failure is tracked to error tracking and the request is unaffected (verified by unit spec and GDK smoke test). - **Sampling granularity is global.** Percentage-of-actors applies per request across all fleets; there is no per-fleet targeting. Getting a useful sample of the low-rate ai-assisted divergence (~0.2/s) requires a higher global percentage for a few hours, which oversamples the high-rate classes at the same time. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post results to the team channel. **Non-production** (verify line shape in logs before production): ``` /chatops gitlab run feature set log_labkit_rack_divergence true --staging ``` **Production** (percentage of actors; each step is a bounded diagnosis window, not a ratchet): ``` /chatops gitlab run feature set log_labkit_rack_divergence 1 --actors ``` 1. Start at 1%, confirm line rate and shape in Kibana (`json.message: "Labkit rack shadow divergence"`). 2. Raise to 25-100% for a bounded window (hours) to capture the ai-assisted and api identity samples needed for https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362. 3. Return to `false` between diagnosis windows: ``` /chatops gitlab run feature set log_labkit_rack_divergence false ``` ## Cleanup The flag and the log line it gates are removed together once the divergence classes have documented root causes and dispositions (the diagnosis-plan exit criteria in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362). Remove the flag, the `log_divergence` path in `lib/gitlab/rack_attack/labkit_rate_limit/divergence.rb`, and the YAML definition, then: ``` /chatops gitlab run feature delete log_labkit_rack_divergence --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set log_labkit_rack_divergence false ```