0

I created K3S master on AWS with Ubuntu 18.04 as follows:

K3S MASTER SERVER IP=54.252.228.96

ubuntu@ip-10-0-0-62:~$ curl -sfL https://get.k3s.io | sh -
sudo kubectl get nodes
ip-10-0-0-62   Ready    master   11s   v1.18.8+k3s1

I created another node on Azure and installed the K3S agent as below

azureuser@k3-worker-node-01:~$ export NODE_TOKEN=K10e5ae49a964b013769ddd64f6541fc89de8eae6ec18ae5a6cdb432dfc59146226::server:b08cc8be4a07986ac1d062e9061ca74b
azureuser@k3-worker-node-01:~$ curl -sfL https://get.k3s.io | K3S_URL=https://54.252.228.96:6443 K3S_TOKEN=$NODE_TOKEN sh -

K3S Worker Node IP=52.243.64.166

More details about Azure worker node

root@k3s-worker-node-01:/var/lib/rancher/k3s/agent# hostname
k3s-worker-node-01
root@k3s-worker-node-01:/var/lib/rancher/k3s/agent# curl ifconfig.io
52.243.64.166

However, I do not see master showing worker node (Azure)... Do NOTE: I can see my node from AWS join to AWS K3S Master!

ubuntu@ip-10-0-0-62:~$ sudo kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
ip-10-0-0-45   NotReady   <none>   99m     v1.18.8+k3s1
ip-10-0-0-62   Ready      master   4h27m   v1.18.8+k3s1
ip-10-0-0-52   Ready      <none>   136m    v1.18.8+k3s1

Logs on Azure Worker node

**/var/log/syslog**
Sep 14 16:54:29 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:54:29.923739251Z" level=info msg="Connecting to proxy" url="wss://10.0.0.62:6443/v1-k3s/connect"
Sep 14 16:56:40 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:56:40.091361388Z" level=error msg="Failed to connect to proxy" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 16:56:40 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:56:40.091404188Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 16:56:45 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:56:45.091592445Z" level=info msg="Connecting to proxy" url="wss://10.0.0.62:6443/v1-k3s/connect"
Sep 14 16:58:55 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:58:55.259345192Z" level=error msg="Failed to connect to proxy" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 16:58:55 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:58:55.259417391Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 16:59:00 k3s-worker-node-01 k3s[2279]: time="2020-09-14T16:59:00.259623743Z" level=info msg="Connecting to proxy" url="wss://10.0.0.62:6443/v1-k3s/connect"
Sep 14 17:01:10 k3s-worker-node-01 k3s[2279]: time="2020-09-14T17:01:10.427330511Z" level=error msg="Failed to connect to proxy" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 17:01:10 k3s-worker-node-01 k3s[2279]: time="2020-09-14T17:01:10.427378010Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.0.62:6443: connect: connection timed out"
Sep 14 17:01:15 k3s-worker-node-01 k3s[2279]: time="2020-09-14T17:01:15.427601475Z" level=info msg="Connecting to proxy" url="wss://10.0.0.62:6443/v1-k3s/connect"

The Network Security Group on Azure is as below: enter image description here Outbound NSG: enter image description here

The Security group on AWS K3S Master is like below: Inbound

enter image description here

Outbound enter image description here

Please assist me as to why I am unable to get the basic stuff working.

P.S: Thanks Marat for your persistence!

AWS Worker Node which is working

ubuntu@ip-10-0-0-52:~$ sudo su -
root@ip-10-0-0-52:~# ss -tulnp
Netid     State       Recv-Q      Send-Q             Local Address:Port              Peer Address:Port
udp       UNCONN      0           0                  127.0.0.53%lo:53                     0.0.0.0:*          users:(("systemd-resolve",pid=918,fd=12))
udp       UNCONN      0           0                 10.0.0.52%eth0:68                     0.0.0.0:*          users:(("systemd-network",pid=894,fd=17))
udp       UNCONN      0           0                        0.0.0.0:8472                   0.0.0.0:*
tcp       LISTEN      0           128                    127.0.0.1:10249                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=136))
tcp       LISTEN      0           128                      0.0.0.0:31760                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=127))
tcp       LISTEN      0           128                    127.0.0.1:10256                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=133))
tcp       LISTEN      0           128                127.0.0.53%lo:53                     0.0.0.0:*          users:(("systemd-resolve",pid=918,fd=13))
tcp       LISTEN      0           128                      0.0.0.0:22                     0.0.0.0:*          users:(("sshd",pid=1301,fd=3))
tcp       LISTEN      0           128                    127.0.0.1:44793                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=5))
tcp       LISTEN      0           128                    127.0.0.1:10010                  0.0.0.0:*          users:(("containerd",pid=2429,fd=9))
tcp       LISTEN      0           128                    127.0.0.1:6010                   0.0.0.0:*          users:(("sshd",pid=2197,fd=10))
tcp       LISTEN      0           128                      0.0.0.0:31838                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=126))
tcp       LISTEN      0           128                    127.0.0.1:10248                  0.0.0.0:*          users:(("k3s-agent",pid=2393,fd=55))
tcp       LISTEN      0           128                            *:10250                        *:*          users:(("k3s-agent",pid=2393,fd=61))
tcp       LISTEN      0           128                         [::]:22                        [::]:*          users:(("sshd",pid=1301,fd=4))
tcp       LISTEN      0           128                        [::1]:6010                      [::]:*          users:(("sshd",pid=2197,fd=9))

Azure Worker Node which is not working

root@k3s-worker-node-01:~# ss -tulnp
Netid     State       Recv-Q      Send-Q              Local Address:Port             Peer Address:Port
udp       UNCONN      0           0                   127.0.0.53%lo:53                    0.0.0.0:*          users:(("systemd-resolve",pid=872,fd=12))
udp       UNCONN      0           0                 172.16.0.6%eth0:68                    0.0.0.0:*          users:(("systemd-network",pid=842,fd=19))
tcp       LISTEN      0           4096                    127.0.0.1:45177                 0.0.0.0:*          users:(("k3s-agent",pid=2279,fd=5))
tcp       LISTEN      0           4096                    127.0.0.1:10010                 0.0.0.0:*          users:(("containerd",pid=2306,fd=9))
tcp       LISTEN      0           128                     127.0.0.1:6010                  0.0.0.0:*          users:(("sshd",pid=2059,fd=10))
tcp       LISTEN      0           128                 127.0.0.53%lo:53                    0.0.0.0:*          users:(("systemd-resolve",pid=872,fd=13))
tcp       LISTEN      0           128                       0.0.0.0:22                    0.0.0.0:*          users:(("sshd",pid=1471,fd=3))
tcp       LISTEN      0           128                         [::1]:6010                     [::]:*          users:(("sshd",pid=2059,fd=9))
tcp       LISTEN      0           128                          [::]:22                       [::]:*          users:(("sshd",pid=1471,fd=4))
root@k3s-worker-node-01:~#

learner
  • 101
  • 3
  • I am not so familiar with k3s, but are you sure that your apiserver is listening on the right address? Additionally, make sure necessary ports are available on the both cloud providers. Based on errors from worker node, you might try to provide CA.crt explicitly or recheck that firewall rules for worker node on Azure. – Marat Gainutdinov Sep 14 '20 at 13:40
  • What container runtime do you use Docker or containerd? make sure all system-critical containers are up and running on master node. – Marat Gainutdinov Sep 14 '20 at 13:48
  • K3S Master is fine. I can connect to worker node from AWS itself, when I try to connect from Azure that is where the issue arises. – learner Sep 14 '20 at 15:31
  • can you provide ss -tulnp # output from both nodes – Marat Gainutdinov Sep 14 '20 at 17:19
  • Hi Marat, Updated the question with the feedback you requested. – learner Sep 14 '20 at 17:34
  • kubectl get nodes -o wide # could you please provide this as well, based on output I assume the control plane endpoint's is not reachable from worker node or apiserver listening on another port for incoming connection. – Marat Gainutdinov Sep 15 '20 at 12:38

0 Answers0