## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/606107) currently behind the `semantic_search_indexing_set_root_namespace_id` feature flag. - DRI: @<gitlab-username-of-dri> - Team Slack channel: `#g_ai_core_infra` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? Enabling this FF will introduce a major change in the "Bulk Embeddings Indexing" stage of the Semantic Search Indexing pipeline. This has been tested thoroughly, so we do not expect any issues. If an error does occur, items that failed processing will be retried through an existing retry mechanism. If that still fails, items will be moved to a DeadQueue, which can be replayed to re-process the items. No data loss is expected. ## Dashboards and Logs to monitor - [Error log](https://log.gprd.gitlab.net/app/r/s/qPZ2X) - [Embeddings generation log](https://log.gprd.gitlab.net/app/r/s/IvcLx) - [Bulk embeddings indexing completion log](https://log.gprd.gitlab.net/app/r/s/NPvEC) - [Bulk embeddings indexing duration visualization - partitioned by duration range](https://log.gprd.gitlab.net/app/r/s/xxkat) - - [Bulk embeddings indexing duration visualization - partitioned by duration per reference range](https://log.gprd.gitlab.net/app/r/s/xLywz) ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_ai_core_infra`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set semantic_search_indexing_set_root_namespace_id 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set semantic_search_indexing_set_root_namespace_id true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set semantic_search_indexing_set_root_namespace_id <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss semantic_search_indexing_set_root_namespace_id true /chatops gitlab run feature set --group=gitlab-org,gitlab-com semantic_search_indexing_set_root_namespace_id true /chatops gitlab run feature set --user=<gitlab-username-of-dri> semantic_search_indexing_set_root_namespace_id true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246906 19.3 /chatops gitlab run feature delete semantic_search_indexing_set_root_namespace_id --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set semantic_search_indexing_set_root_namespace_id false # production /chatops gitlab run feature set semantic_search_indexing_set_root_namespace_id false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete semantic_search_indexing_set_root_namespace_id --dev --pre --staging --staging-ref --production # remove entirely ```