I'm trying to create an HA k3s cluster using Proxmox and a small fleet of Raspberriy PIs 4B.
For the PIs everything works fine, but when trying to install a master on a Proxmox VM it will not start.
My Setup:
- Host: Proxmox 7.0.7 (I tried with 6.4.4 as well)
- Guest: Ubuntu 20.04.2
- K3S: v1.21.3+k3s1 (I tried with v1.19.13+k3s1 as well)
- MariaDB: 10.3
I'm running these commands in order to install the master
export K3S_DATASTORE_ENDPOINT='mysql://DB_USER:DB_PASSWORD@tcp(DB_IP:DB_PORT)/DB_SCHEME'
curl -sfL https://get.k3s.io | sh -s - server --node-taint CriticalAddonsOnly=true:NoExecute --tls-san NGINX_LOADBALANCER_IP
This is the output of installation and startup:
[INFO] Using v1.19.13+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.19.1 3+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.19 .13+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.
I checked the logs:
systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2021-08-03 20:27:40 UTC; 2s ago
Docs: https://k3s.io
Process: 6181 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 6193 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 6194 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 6195 ExecStart=/usr/local/bin/k3s server --node-taint CriticalAddonsOnly=true:NoExecute --tls-san NGINX_LOADBALANNCER_IP (code=exited, status=1/FAILURE)
Main PID: 6195 (code=exited, status=1/FAILURE)
Aug 03 20:27:40 k3svm1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 20:27:40 k3svm1 systemd[1]: k3s.service: Failed with result 'exit-code'.
Aug 03 20:27:40 k3svm1 systemd[1]: Failed to start Lightweight Kubernetes.
and:
journalctl -u k3s.service
Aug 03 20:06:17 k3svm1 systemd[1]: Starting Lightweight Kubernetes...
Aug 03 20:06:17 k3svm1 sh[19450]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Aug 03 20:06:17 k3svm1 sh[19451]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Aug 03 20:06:17 k3svm1 k3s[19460]: time="2021-08-03T20:06:17Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Aug 03 20:06:17 k3svm1 k3s[19460]: time="2021-08-03T20:06:17Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.748397535Z" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.751745749Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.751876220Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.753348552Z" level=info msg="Database tables and indexes are up to date"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.757714719Z" level=info msg="Kine listening on unix://kine.sock"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.764631916Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.765377675Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.766187231Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.766815165Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.767415198Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 >
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.767950031Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.768698847Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:0>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.769745716Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.770870630Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:1>
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.771882180Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.772508382Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.773399505Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.813171353Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1628021178: notBefore=2021-08-03 20:06:18 +0000 UTC notAfter=2022-08-03 20:06:18 +0000 UTC"
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.813556476Z" level=info msg="Active TLS secret (ver=) (count 9): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn->
Aug 03 20:06:18 k3svm1 k3s[19460]: time="2021-08-03T20:06:18.819862032Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Aug 03 20:06:18 k3svm1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 20:06:18 k3svm1 systemd[1]: k3s.service: Failed with result 'exit-code'.
Aug 03 20:06:18 k3svm1 systemd[1]: Failed to start Lightweight Kubernetes.
Aug 03 20:06:23 k3svm1 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
Aug 03 20:06:23 k3svm1 systemd[1]: Stopped Lightweight Kubernetes.
Aug 03 20:06:23 k3svm1 systemd[1]: Starting Lightweight Kubernetes...
Aug 03 20:06:23 k3svm1 sh[19478]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Aug 03 20:06:23 k3svm1 sh[19483]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.115279840Z" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.119390931Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.119554649Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.121305745Z" level=info msg="Database tables and indexes are up to date"
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.125898745Z" level=info msg="Kine listening on unix://kine.sock"
Aug 03 20:06:24 k3svm1 k3s[19489]: time="2021-08-03T20:06:24.146164308Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Aug 03 20:06:24 k3svm1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 20:06:24 k3svm1 systemd[1]: k3s.service: Failed with result 'exit-code'.
Aug 03 20:06:24 k3svm1 systemd[1]: Failed to start Lightweight Kubernetes.
Aug 03 20:06:29 k3svm1 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 2.
Aug 03 20:06:29 k3svm1 systemd[1]: Stopped Lightweight Kubernetes.
Aug 03 20:06:29 k3svm1 systemd[1]: Starting Lightweight Kubernetes...
Aug 03 20:06:29 k3svm1 sh[19507]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Aug 03 20:06:29 k3svm1 sh[19508]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.565328025Z" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.568959518Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.568994906Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.570693830Z" level=info msg="Database tables and indexes are up to date"
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.575194321Z" level=info msg="Kine listening on unix://kine.sock"
Aug 03 20:06:29 k3svm1 k3s[19511]: time="2021-08-03T20:06:29.594809727Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Aug 03 20:06:29 k3svm1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 20:06:29 k3svm1 systemd[1]: k3s.service: Failed with result 'exit-code'.
Aug 03 20:06:29 k3svm1 systemd[1]: Failed to start Lightweight Kubernetes.
Aug 03 20:06:34 k3svm1 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 3.
Aug 03 20:06:34 k3svm1 systemd[1]: Stopped Lightweight Kubernetes.
Aug 03 20:06:34 k3svm1 systemd[1]: Starting Lightweight Kubernetes...
Aug 03 20:06:34 k3svm1 sh[19527]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Aug 03 20:06:34 k3svm1 sh[19530]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
From there the restarts continue.
As mentioned I've tried with multiple versions, but nothing seems to work. Also I am not really getting an error. The only hint I've found in different GitHub issues was to enable containerization regarding Raspberry PI by editing /boot/cmdline.txt. However I am not getting the issue on a PI but rather on a Proxmox-VM.
Is there something I am missing?
Somehow this guy managed to get it to run in the same setup. Did someone else make it run and might provide some reference?
