1

I have GitLab installed in Kubernetes with their Helm chart.

I migrated my old Gitlab deployment from one cluster to another with the following steps:

  • Scale down all pods in old cluster
  • Apply values.yml with helm to new cluster (to create PVCs)
  • Scale down all pods in new cluster
  • Change DNS records, HAProxy, etc
  • Manually rsync data from old PVCs to new PVCs (minio, gitaly, redis, postgres, prometheus)
  • Run helm upgrade to bring deployments back online in new cluster

After all that the deployment for the most part works fine. Able to login and use git.

But the runner is failing to register, so I can't run any CI. Looking at the gitlab-gitlab-runner pod, I see the message below repeated over and over:

Registration attempt 30 of 30
Runtime platform                                    arch=amd64 os=linux pid=691 revision=3b6f852e version=14.0.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
 
ERROR: Registering runner... failed                 runner=y6ixJoR1 status=500 Internal Server Error
PANIC: Failed to register the runner. You may be having network problems.

As you can see, it's failing to register the runner. Trying to go to /admin/runners gives me a 500 error.

Where can I see more information as to why I am getting this 500 error?

Mikołaj Głodziak
  • 323
  • 1
  • 10
cclloyd
  • 583
  • 1
  • 13
  • 24

2 Answers2

1

It looks like you have restored data and configuration from the previous cluster, however you are missing the secrets on the new cluster. Here you can find a procedure, what should you do, when the secrets are lost. You can also find there:

In cases like CI/CD variables and runner authentication, you can experience unexpected behaviors, such as:

  • Stuck jobs.
  • 500 errors. #you got this error

See also similar problems:

Mikołaj Głodziak
  • 323
  • 1
  • 10
  • 1
    This worked for me. Specifically, resetting the CI vars, runner tokens, project integrations. Though after I still had to manually go to the runners admin page, generate a new registration token, and paste that into the runner secret. After that I deleted the runner pod and it registered just fine. Thanks! – cclloyd Jul 13 '21 at 20:34
0

Q: Trying to go to /admin/runners gives me a 500 error.

A: Assure you've restored /etc/gitlab/gitlab-secrets.json file.

Q: Where can I see more information as to why I am getting this 500 error?

A: It's in /var/log/gitlab/gitlab-rails/production.log

Paul
  • 2,755
  • 6
  • 24
  • 35
Dr. Alexander
  • 101
  • 3