Partner Token Verification - Feature Flag and Rollout Plan

## Summary This issue is to roll out [partner token verification](https://gitlab.com/groups/gitlab-org/-/epics/16890) on production, currently behind the `secret_detection_partner_token_verification` feature flag. ## Owners - Slack channel: `#g_ast-secret-detection` - DRI: @atiwari71 ## Expectations ### What are we expecting to happen? Partner APIs (AWS, GCP, Postman) will verify if detected tokens are valid/active, providing users with accurate token validity status. ### What can go wrong and how would we detect it? - **Partner API errors**: High error rates (>10%) or network failures - **Performance degradation**: P95 latency >5s causing timeouts - **Rate limit exhaustion**: Hitting partner API limits **Detection:** - [Dashboard](https://dashboards.gitlab.net/goto/cf3zed4vmgo3ke?orgId=1) - Alerts: `SecretDetectionPartnerAPIHighErrorRate`, `SecretDetectionPartnerAPIHighLatency` ## Rollout Steps ### Rollout on non-production environments - [x] **19 Nov**: Enable globally on non-production environments ``` /chatops run feature set secret_detection_partner_token_verification true --dev --pre --staging --staging-ref ``` - [ ] Verify feature works in `staging-canary` - [ ] Monitor error rates and latency on dashboard ### Specific rollout on production Execute in `#production`, cross-post to `#g_ast-secret-detection`: - [x] **20 Nov**: Enable for GitLab.org ``` /chatops run feature set --group=gitlab-org,gitlab-com secret_detection_partner_token_verification true ``` - [ ] Verify feature works for GitLab.org projects - [ ] Monitor dashboard for 24 hours ### Global rollout on production - [x] **1 Dec**: Enable globally on GitLab.com ``` /chatops run feature set secret_detection_partner_token_verification true ``` - [ ] Monitor for at least 24 hours ### Release the feature - [ ] **8 Dec**: Create MR to enable feature flag by default ### Remove FF - [ ] **12 Jan**: Create MR to remove feature flag - Remove all code references - Remove YAML definition - [ ] After merge and deploy, delete flag: ``` /chatops run feature delete secret_detection_partner_token_verification --dev --pre --staging --staging-ref --production ``` - [ ] Close this issue ## Rollback Steps - [ ] Disable on production: ``` /chatops run feature set secret_detection_partner_token_verification false ``` - [ ] Disable on non-production: ``` /chatops run feature set secret_detection_partner_token_verification false --dev --pre --staging --staging-ref ```

issue