What happened?
I created a manual script to create all the certs located in /etc/kubernetes/pki and /etc/kubernetes/pki/etcd on a running cluster on centos 7, as well as modifieng the conf files located in /etc/kubernetes {admin.conf, scheduler.conf, controller-manager.conf} and provided new certs for them as well of course without touching kubelet.conf.
I used the ca.crt that was already in /etc/kubernetes/pki and made sure that every cert is created by the ca that is relevant to it.
For example, front-proxy-client.crt is created by front-proxy-ca.crt with this commands:
openssl genrsa -out front-proxy-client.key 2048
openssl req -new -key front-proxy-client.key -out front-proxy-client.csr -config front-proxy-client.conf
openssl x509 -req -in /some-dir/front-proxy-client.csr -CA /etc/kubernetes/pki/front-proxy-ca.crt -CAkey /etc/kubernetes/pki/front-proxy-ca.key -CAcreateserial -out front-proxy-client.crt -days 3500 -extensions v3_ext -extfile front-proxy-client.conf
Taken from this documentation: https://kubernetes.io/docs/tasks/administer-cluster/certificates/#openssl
After running the script and restarting kubelet (moving /etc/kubernetes/manifests folder to another location, waiting 20 seconds and moving it back to its original location), then the cluster didn't came back and 2 things occured,
- When I run
kubectlI get this errorThe connection to the server shiran-va:6443 was refused - did you specify the right host or port? kubeletgot this error
{Name:"<HOSTNAME>.16faefd41a98a537", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"<HOSTNAME>", UID:"<HOSTNAME>", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node <HOSTNAME> status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"<HOSTNAME>"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc0a4df3cd01de737, ext:16891526681, loc:(*time.Location)(0x74fdb40)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc0a4df3cddf6963e, ext:17123831058, loc:(*time.Location)(0x74fdb40)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Patch "https://shiran-va:6443/api/v1/namespaces/default/events/<HOSTNAME>.16faefd41a98a537": dial tcp 127.0.0.1:6443: connect: connection refused'(may retry after sleeping)
Jun 22 08:26:47 <HOSTNAME> kubelet: E0622 08:26:47.059407 1275 kubelet.go:2291] "Error getting node" err="node \"<HOSTNAME>\" not found"
And when running docker ps I can see only 6 containers.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8dab8f31c759 0369cf4303ff "etcd --advertise-cl…" 3 minutes ago Up 3 minutes k8s_etcd_etcd-<HOSTNAME>_kube-system_47193d3eb45a6af6c767d361f5df163b_24
b7941246bdbb k8s.gcr.io/pause:3.4.1 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-apiserver-<HOSTNAME>_kube-system_041857bfd8b1ee80a15a7929c951db88_2
4ab72d5bd333 7a37590177f7 "kube-controller-man…" 2 hours ago Up 2 hours k8s_kube-controller-manager_kube-controller-manager-<HOSTNAME>_kube-system_7e852d8efb7e5c519ccef7c99fff7d77_3
abd6e9b68bea k8s.gcr.io/pause:3.4.1 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-controller-manager-<HOSTNAME>_kube-system_7e852d8efb7e5c519ccef7c99fff7d77_2
2a2fb116728f c67c2461177d "kube-scheduler --au…" 2 hours ago Up 2 hours k8s_kube-scheduler_kube-scheduler-<HOSTNAME>_kube-system_69da6f38673e93a8dcbb9683f1c52166_3
ee8d8f0b7925 k8s.gcr.io/pause:3.4.1 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-scheduler-<HOSTNAME>_kube-system_69da6f38673e93a8dcbb9683f1c52166_2
b6dab97a2306 k8s.gcr.io/pause:3.4.1 "/pause" 2 hours ago Up 2 hours k8s_POD_etcd-<HOSTNAME>_kube-system_47193d3eb45a6af6c767d361f5df163b_2
These are the logs from 8dab8f31c759:
2022-06-22 14:25:45.489024 I | embed: rejected connection from "10.130.1.98:56396" (error "tls: \"10.130.1.98\" does not match any of DNSNames [\"<HOSTNAME>\" \"localhost\"] (lookup <HOSTNAME> on 192.168.11.5:53: server misbehaving)", ServerName "", IPAddresses ["<NODE-IP>" "127.0.0.1" "::1"], DNSNames ["<HOSTNAME>" "localhost"])
These are the logs from 4ab72d5bd333:
E0622 14:23:26.523624 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Get "https://<NODE-IP>:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp <NODE-IP>:6443: connect: connection refused
These are the logs from 2a2fb116728f:
E0622 14:30:28.331655 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://<NODE-IP>:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp <NODE-IP>:6443: connect: connection refused
E0622 14:30:28.688912 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://<NODE-IP>:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp <NODE-IP>:6443: connect: connection refused
What did you expect to happen?
To create a 9 year certs that will replace the ones that comes with the default kubeadm certs renew commnad/kubeadm init.
How can we reproduce it (as minimally and precisely as possible)?
Run these commands:
openssl genrsa -out front-proxy-client.key 2048
openssl req -new -key front-proxy-client.key -out front-proxy-client.csr -config front-proxy-client.conf
openssl x509 -req -in $CURRENT_DIR/front-proxy-client.csr -CA /etc/kubernetes/pki/front-proxy-ca.crt -CAkey /etc/kubernetes/pki/front-proxy-ca.key -CAcreateserial -out front-proxy-client.crt -days 3500 -extensions v3_ext -extfile front-proxy-client.conf
With this conf file:
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
O = system:masters
CN = front-proxy-client
[ v3_ext ]
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=clientAuth
authorityKeyIdentifier=keyid
And place it inside /etc/kubernetes/pki as an example.
I could provide my entire script and x509 config files for this if needed
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:41:19Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
On prem
OS version
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Install tools
Standard kubernetes cluster
Container runtime (CRI) and version (if applicable)
Docker
Edit:
I just noticed that it's creating mainly on apiserver.crt
This is the cert that get created by kubeadm init:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3801787449259067400 (0x34c2a98ad8a00808)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Jun 28 06:50:49 2022 GMT
Not After : Jun 28 06:51:32 2023 GMT
Subject: CN = kube-apiserver
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:f8:09:3f:2d:70:ce:41:3f:6d:ae:dc:5c:5f:b2:
dd:c3:33:18:46:25:e9:fd:9c:b3:d4:19:28:a4:60:
da:14:e7:45:b1:ae:9e:a4:ae:3d:1f:0f:70:db:e8:
55:77:df:56:60:29:c5:18:af:d9:9c:c3:48:11:3e:
68:26:d5:f9:f3:d6:66:39:f7:b0:cb:cf:5d:80:4d:
b0:ce:5b:e5:98:a1:86:81:d2:98:67:77:bd:f0:94:
a5:e7:07:49:4c:f9:1c:37:26:56:4c:9f:ff:7b:f6:
98:e8:a0:55:7e:86:95:ab:b8:e5:c9:2b:75:15:56:
21:d1:af:00:33:ec:de:37:68:66:9d:2e:3c:08:35:
25:5b:d8:a3:53:6e:96:de:30:f0:54:15:ec:74:3c:
7d:52:91:55:b0:24:d4:32:52:7d:a0:13:d0:67:95:
b5:4e:2d:e1:c3:d8:0f:72:23:b4:4b:8f:c4:89:59:
21:ac:ac:00:22:b4:d1:1d:1d:b4:1e:2a:7f:86:0e:
ab:4a:e8:32:62:87:e7:65:06:7c:b3:6c:b7:39:e7:
12:9c:af:43:e9:41:a0:76:be:5f:d3:0c:f4:6d:a6:
d4:87:3c:22:84:22:d9:71:37:6f:1d:0b:b5:a3:df:
4f:c2:9b:15:c5:f0:77:27:e6:49:fa:e6:c0:af:9a:
27:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:A8:5E:18:DD:85:68:FC:69:91:D8:11:28:BB:50:76:E4:7D:43:D8:70
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:<fqdn>, DNS:<hostname>, IP Address:<unknown>, IP Address:<node-ip>
Signature Algorithm: sha256WithRSAEncryption
d1:b9:4f:fa:6d:05:6c:2d:65:62:01:c0:39:cd:e4:ee:03:56:
25:cc:0f:df:ed:56:26:0d:a6:4e:d4:ef:56:33:66:7d:2c:55:
07:68:a0:18:2b:13:ca:7e:f9:cb:d8:2d:ec:df:94:f3:2c:20:
ed:f9:5a:a5:c2:5b:8a:77:56:e8:99:20:bf:d4:65:b7:5b:9b:
2e:4f:f3:2e:83:a8:ae:2d:4a:e9:f2:dd:bb:65:e6:cb:0b:c5:
9b:09:b6:56:e6:58:2a:f6:89:2f:9a:e9:24:6d:db:92:ae:aa:
49:19:27:c1:03:6e:74:55:92:e2:a0:73:29:49:f6:b4:0f:12:
81:33:9b:bc:9f:92:c4:4a:37:4a:4e:32:8d:1e:74:67:e3:00:
40:46:a9:4a:6b:b1:32:bf:4d:a9:c5:8c:ff:80:a8:3a:ac:0f:
d8:1d:62:8a:e0:3a:1a:8d:cf:79:65:c6:20:be:6b:8b:85:a8:
74:25:8d:c2:80:62:fc:07:c5:87:89:cf:60:6f:80:e0:68:bb:
4c:ee:52:be:2e:eb:66:f9:bd:19:04:f9:6c:f9:2f:4c:b2:4a:
ff:de:bd:76:c7:49:6e:4f:e3:58:20:3a:37:d4:db:4b:97:5a:
b1:49:75:5d:9f:8f:ad:0f:a8:d3:76:96:bb:67:43:c4:dd:92:
bf:21:75:93
-----BEGIN CERTIFICATE-----
MIIDmDCCAoCgAwIBAgIINMKpitigCAgwDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yMjA2MjgwNjUwNDlaFw0yMzA2MjgwNjUxMzJaMBkx
FzAVBgNVBAMTDmt1YmUtYXBpc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEA+Ak/LXDOQT9trtxcX7LdwzMYRiXp/Zyz1BkopGDaFOdFsa6epK49
Hw9w2+hVd99WYCnFGK/ZnMNIET5oJtX589ZmOfewy89dgE2wzlvlmKGGgdKYZ3e9
8JSl5wdJTPkcNyZWTJ//e/aY6KBVfoaVq7jlySt1FVYh0a8AM+zeN2hmnS48CDUl
W9ijU26W3jDwVBXsdDx9UpFVsCTUMlJ9oBPQZ5W1Ti3hw9gPciO0S4/EiVkhrKwA
IrTRHR20Hip/hg6rSugyYofnZQZ8s2y3OecSnK9D6UGgdr5f0wz0babUhzwihCLZ
cTdvHQu1o99PwpsVxfB3J+ZJ+ubAr5onzQIDAQABo4HnMIHkMA4GA1UdDwEB/wQE
AwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQY
MBaAFKheGN2FaPxpkdgRKLtQduR9Q9hwMIGNBgNVHREEgYUwgYKCCmt1YmVybmV0
ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5kZWZhdWx0LnN2Y4Ik
a3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsgglzaGlyYW4tdmGC
C3NoaXJhbi12YS0yhwQKYAABhwQKggFhMA0GCSqGSIb3DQEBCwUAA4IBAQDRuU/6
bQVsLWViAcA5zeTuA1YlzA/f7VYmDaZO1O9WM2Z9LFUHaKAYKxPKfvnL2C3s35Tz
LCDt+VqlwluKd1bomSC/1GW3W5suT/Mug6iuLUrp8t27ZebLC8WbCbZW5lgq9okv
mukkbduSrqpJGSfBA250VZLioHMpSfa0DxKBM5u8n5LESjdKTjKNHnRn4wBARqlK
a7Eyv02pxYz/gKg6rA/YHWKK4Doajc95ZcYgvmuLhah0JY3CgGL8B8WHic9gb4Dg
aLtM7lK+Lutm+b0ZBPls+S9Mskr/3r12x0luT+NYIDo31NtLl1qxSXVdn4+tD6jT
dpa7Z0PE3ZK/IXWT
-----END CERTIFICATE-----
And this is the one I created:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:46:c6:e4:82:da:a6:6a:e4:a5:be:1b:25:51:ff:0e:d2:64:ca:83
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Jun 28 09:51:45 2022 GMT
Not After : Jan 27 09:51:45 2032 GMT
Subject: CN = kube-apiserver
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:9c:d6:2e:ed:a4:ab:4e:e4:da:41:9e:6b:e5:c3:
6c:34:59:75:13:bd:4f:a2:9b:2a:bc:17:56:dc:7e:
b8:a5:88:81:0b:22:11:7f:08:09:e1:29:8c:02:68:
28:06:7f:f6:28:e5:dc:24:42:7d:83:d8:8e:72:78:
58:6c:b4:2d:67:72:ef:36:95:97:f8:ba:70:c2:13:
2c:c2:68:f7:91:57:ce:84:f4:e4:8a:f1:76:48:aa:
27:05:3c:3d:24:ce:21:95:d1:e7:ae:be:64:ee:e0:
62:82:0f:e9:d8:ff:75:7e:27:54:d5:f0:18:61:95:
c6:6e:f3:87:47:7f:6a:a9:b5:34:f5:42:fc:2d:ef:
9a:9e:73:ce:03:aa:e7:38:da:3c:5e:19:97:2e:9a:
67:18:ab:c7:bb:36:54:28:d7:0f:28:05:37:bf:4f:
83:ac:54:8a:8d:f1:44:65:60:91:6b:43:a7:21:02:
6e:43:b1:8b:90:66:59:1b:cc:9a:ef:62:81:54:8a:
12:b8:98:8e:e3:17:a4:0f:f7:3b:e2:eb:cb:a6:5c:
e7:0f:bb:04:91:f3:7d:cf:ed:48:7f:44:24:cc:ce:
3c:34:67:4a:4b:19:52:9d:5e:c1:5b:35:d9:23:2f:
c5:11:36:4e:d7:b9:f3:84:db:82:8b:35:a3:1d:ac:
4b:15
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:<hostname>, DNS:<fqdn>, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:<unknown>, IP Address:<node-ip>
X509v3 Authority Key Identifier:
keyid:A8:5E:18:DD:85:68:FC:69:91:D8:11:28:BB:50:76:E4:7D:43:D8:70
Signature Algorithm: sha256WithRSAEncryption
e0:8c:cf:0e:8e:9f:fe:9b:a4:de:a4:c4:39:9b:1c:c8:81:a9:
54:fb:d1:3d:b7:44:01:8c:40:da:7e:53:4b:5d:a0:49:32:54:
bf:67:6e:86:c2:21:df:d2:b1:68:7a:f1:9b:a8:ff:1f:c7:79:
9f:b4:87:6b:3e:e4:d2:f2:a1:73:52:0e:1b:84:4c:9b:18:da:
ed:c8:7a:06:83:47:4c:a7:ba:d3:b2:47:b8:74:25:8a:db:79:
75:98:16:9f:9f:11:83:96:5e:51:d9:d2:fe:04:9f:ef:4e:2f:
52:89:cd:76:71:35:f9:47:2b:40:0c:c6:43:6e:84:ba:55:38:
e4:fc:94:df:b3:7a:81:10:04:0c:53:7d:85:43:84:a0:cf:b0:
11:d2:66:7f:90:ad:02:a7:a5:97:c2:28:fc:2a:13:2b:0e:a6:
74:1b:68:3b:c7:bf:b8:c3:63:eb:c7:fb:f2:b0:f5:13:2b:4a:
ba:6f:9c:d5:fc:76:ec:f4:f5:c1:b5:ec:5b:08:22:a4:8d:39:
4e:96:5c:86:5f:66:a5:7e:b7:7e:3e:60:85:0f:27:ff:4d:57:
ed:c2:da:65:cb:36:b3:f2:33:4a:34:3d:f5:f6:26:ba:4e:36:
83:2c:85:29:8b:25:aa:d3:1a:ca:50:a2:a3:57:0f:84:f9:f8:
32:c5:8f:d9
-----BEGIN CERTIFICATE-----
MIIDpDCCAoygAwIBAgIUREbG5ILapmrkpb4bJVH/DtJkyoMwDQYJKoZIhvcNAQEL
BQAwFTETMBEGA1UEAxMKa3ViZXJuZXRlczAeFw0yMjA2MjgwOTUxNDVaFw0zMjAx
MjcwOTUxNDVaMBkxFzAVBgNVBAMTDmt1YmUtYXBpc2VydmVyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnNYu7aSrTuTaQZ5r5cNsNFl1E71PopsqvBdW
3H64pYiBCyIRfwgJ4SmMAmgoBn/2KOXcJEJ9g9iOcnhYbLQtZ3LvNpWX+LpwwhMs
wmj3kVfOhPTkivF2SKonBTw9JM4hldHnrr5k7uBigg/p2P91fidU1fAYYZXGbvOH
R39qqbU09UL8Le+annPOA6rnONo8XhmXLppnGKvHuzZUKNcPKAU3v0+DrFSKjfFE
ZWCRa0OnIQJuQ7GLkGZZG8ya72KBVIoSuJiO4xekD/c74uvLplznD7sEkfN9z+1I
f0QkzM48NGdKSxlSnV7BWzXZIy/FETZO17nzhNuCizWjHaxLFQIDAQABo4HnMIHk
MAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF
BwMBMIGNBgNVHREEgYUwgYKCC3NoaXJhbi12YS0xgglzaGlyYW4tdmGCCmt1YmVy
bmV0ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5kZWZhdWx0LnN2
Y4Ika3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FshwQKYAABhwQK
ggFgMB8GA1UdIwQYMBaAFKheGN2FaPxpkdgRKLtQduR9Q9hwMA0GCSqGSIb3DQEB
CwUAA4IBAQDgjM8Ojp/+m6TepMQ5mxzIgalU+9E9t0QBjEDaflNLXaBJMlS/Z26G
wiHf0rFoevGbqP8fx3mftIdrPuTS8qFzUg4bhEybGNrtyHoGg0dMp7rTske4dCWK
23l1mBafnxGDll5R2dL+BJ/vTi9Sic12cTX5RytADMZDboS6VTjk/JTfs3qBEAQM
U32FQ4Sgz7AR0mZ/kK0Cp6WXwij8KhMrDqZ0G2g7x7+4w2Prx/vysPUTK0q6b5zV
/Hbs9PXBtexbCCKkjTlOllyGX2alfrd+PmCFDyf/TVftwtplyzaz8jNKND319ia6
TjaDLIUpiyWq0xrKUKKjVw+E+fgyxY/Z
-----END CERTIFICATE-----
Small Update:
I have generating kubeadm CSRs with the following command:
kubeadm certs generate-csr --cert-dir /some/path/ --kubeconfig-dir /some/path/
And after comparing the admin.conf generated by the command and the admin.conf kubeadm creates when running kubeadm init, then I saw that the server section is different and probably when I create the certs manually (without generate-csr), then I should make sure that they are both the same.
This is the result when I run generate-csr command:
apiVersion: v1
clusters:
- cluster:
server: https://<node-ip>:6443
name: kubernetes
And this is the one created by kubeadm init/kubeadm renew certs admin.conf:
server: https://shiran-va:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
