I created a K3s multi-master embedded cluster like this:
hostname: k3s01
curl -sfL https://get.k3s.io | K3S_TOKEN=xxx INSTALL_K3S_EXEC="server --disable servicelb --disable traefik --bind-address=10.0.0.4 --tls-san 10.0.0.4 --node-external-ip=168.119.x.x --node-ip=10.0.0.4 --flannel-iface=enp7s0 --advertise-address=PUBIP-OF-LB --cluster-init" sh -
hostname: k8s02
curl -sfL https://get.k3s.io | K3S_TOKEN=xxx INSTALL_K3S_EXEC="server --disable servicelb --disable traefik --bind-address=10.0.0.2 --tls-san 10.0.0.2 --node-ip 10.0.0.2 --node-external-ip=168.119.x.x --flannel-iface=enp7s0 --server=https://10.0.0.4:6443" sh -
hostname: k8s03
curl -sfL https://get.k3s.io | K3S_TOKEN=xxx INSTALL_K3S_EXEC="server --disable servicelb --disable traefik --bind-address=10.0.0.3 --tls-san 10.0.0.3 --node-ip 10.0.0.3 --node-external-ip=168.119.x.x --flannel-iface=enp7s0 --server=https://10.0.0.4:6443" sh -
I can connect from my local machine with kubectl via the LB-IP! LB: tcp 6443 -> 6443
I can also use kubectl from within any of the nodes above. I deployed CSI for Hetzner, that also works fine. Tested with their test-deployment!
However, after all of that (working fine so far) i tried to install ingress-nginx. The deployment started without any issue. But i found there is an issue communicating with the apiserver from within the cluster, as the following log of the ingress-nginx-controller shows:
E1204 11:42:25.216392 8 leaderelection.go:321] error retrieving resource lock ingress-nginx/ingress-controller-leader-nginx: Get "https://10.43.0.1:443/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx": dial tcp 10.43.0.1:443: connect: connection refused
Hm, Strange! Ok, let's do some checks:
kubectl get svc kubernetes -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-12-04T11:22:25Z"
labels:
component: apiserver
provider: kubernetes
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:component: {}
f:provider: {}
f:spec:
f:clusterIP: {}
f:ports:
.: {}
k:{"port":443,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:sessionAffinity: {}
f:type: {}
manager: k3s
operation: Update
time: "2020-12-04T11:22:25Z"
name: kubernetes
namespace: default
resourceVersion: "10434"
selfLink: /api/v1/namespaces/default/services/kubernetes
uid: f0993556-3b7f-40aa-a293-45170cb03002
spec:
clusterIP: 10.43.0.1
ports:
- name: https
port: 443
protocol: TCP
targetPort: 6443
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Looks gtm.
kubectl get endpoints -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2020-12-04T11:22:25Z"
labels:
endpointslice.kubernetes.io/skip-mirror: "true"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:endpointslice.kubernetes.io/skip-mirror: {}
f:subsets: {}
manager: k3s
operation: Update
time: "2020-12-04T11:23:39Z"
name: kubernetes
namespace: default
resourceVersion: "808"
selfLink: /api/v1/namespaces/default/endpoints/kubernetes
uid: cb450392-b4c9-4c2f-bfde-1a3b20ac4b5d
subsets:
- addresses:
- ip: 167.233.x.x
- ip: 168.119.x.x
- ip: 168.119.x.x
ports:
- name: https
port: 6443
protocol: TCP
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Ok, why are Pub IPs here? lets check it from within one pod to call one of the IP directly:
kubectl exec -it ingress-controler-pod-xxxx -- bash
bash-5.0$ curl https://167.233.x.x:6443 --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}bash-5.0$ curl https://10.43.0.1:443
curl: (7) Failed to connect to 10.43.0.1 port 443: Connection refused
Ok.. thats strange!
Also sometimes some errors like:
Error from server: error dialing backend: dial tcp: lookup k8s02: Try again
appear when i try top exex into a pod or display logs. That only happens when i try to do this to a destination pod running on another host.
Is there something wrong with DNS?
cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
I cant resolve my hosts by their name. However i have just specified IPs in the K3s Setup. Do i need working DNS between my Hosts? Is there something wrong with my K3s Install Params?
