<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary ~"GitLab Ultimate" [customer](https://gitlab.my.salesforce.com/00161000002xBZQAA2) raised a support [ticket](https://gitlab.zendesk.com/agent/tickets/177069) (internal links) to report that errors occurred if - The runner abruptly restarted for some reason (re-registering after it had restarted) - They upgrade a runner CI Job pods under the previous runner will sometimes finish successfully, sometimes they'll be stuck running for days. Errors include: - `ERROR: Job failed (system failure): Unauthorized"` - `ERROR: Error cleaning up pod: Unauthorized`, `ERROR: Error cleaning up secrets: Unauthorized` (but `Job succeeded`) ### Steps to reproduce Reproduced by @atanayno - install runner via helm - start a long-running job - uninstall runner via helm uninstall - job may complete with success, but there will be errors: ``` ... Cleaning up file based variables ERROR: Error cleaning up pod: Unauthorized ERROR: Error cleaning up secrets: Unauthorized Job succeeded ``` <!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. --> ### Example Project <!-- If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version. --> ### What is the current *bug* behavior? Race condition whereby the uninstall prevents the runner cleaning up the running jobs. @steveazz hypothesised: - Assumption: runners are deployed with RBAC roles and rolebindings created by the GitLab helm chart - `helm uninstall` DELETES the roles and rolebinding, and starts terminating the runner pod - The runner pod carries on running until the job completes - Job finishes, runner tried to delete it. But as the rolebinding was deleted a long time ago (`helm delete`) runner pod doesn’t have access to delete pods/secrets - Result: `Unauthorized` ### What is the expected *correct* behavior? <!-- Describe what you should see instead. --> ### Partial workaround Before upgrading a runner deployed with helm - Pause it in GitLab - Wait for any running jobs to complete - Then perform the upgrade ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. -->