[FF] `snowplow_emitter_http_timeout` -- Add HTTP timeout to Snowplow emitter

## Summary This issue is to roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/562794) on production, that is currently behind the `snowplow_emitter_http_timeout` feature flag. ## Owners - Most appropriate Slack channel to reach out to: `#g_analytics_analytics_instrumentation` - Best individual to reach out to: @michold ## Expectations ### What are we expecting to happen? When enabled, the Snowplow emitter will use a custom `SnowplowTimeoutEmitter` (subclass of `AsyncEmitter`) that sets `open_timeout` and `read_timeout` to 30 seconds on HTTP POST requests. This is to debug staging issues where event emission metrics don't match. ### What can go wrong and how would we detect it? Snowplow events may fail to send if the timeout is too aggressive. Monitor the `gitlab_snowplow_failed_events_total` and `gitlab_snowplow_successful_events_total` counters on https://dashboards.gitlab.net. ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops run auto_deploy status <merge-commit-of-your-feature>` - [ ] Enable the feature globally on non-production environments with `/chatops run feature set snowplow_emitter_http_timeout true --dev --pre --staging --staging-ref` - [ ] Verify that the feature works as expected. - [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). ### Before production rollout - [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. - `/chatops run feature set snowplow_emitter_http_timeout true` - Monitor the appropriate graphs on https://dashboards.gitlab.net. ### Release the feature - [ ] Create a merge request to remove the `snowplow_emitter_http_timeout` feature flag. - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops run feature set snowplow_emitter_http_timeout false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops run feature set snowplow_emitter_http_timeout false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops run feature delete snowplow_emitter_http_timeout --dev --pre --staging --staging-ref --production ```

issue