0

I tried to configure Wireguard for a K3s multi-site cluster, my issue is with routing. There are at least 4 nodes, 2 in one location + 1 in other location and + 1 in other location.

Assumptions:

  • 10.50.0.0/16 network is for physical nodes
  • 10.42.0.0/16 network is for pods
  • 10.43.0.0/16 network is for services
  • Each of physical node can create a Pod, which have assigned an unique IP address in 10.42.0.0/16 range
  • Each Pod on each physical node should be able to ping each other

Deployed configuration worked only between last two nodes, the rest nodes cannot even ping each other. I guess it may be something with the routing, but I have not enough knowledge to check this.

Example configurations from two of nodes:

10.50.54.100:

[Interface]
# primary-1
Address = 10.50.54.100/24
PrivateKey = pr111111111111111111111111111111111111111111
ListenPort = 443
MTU = 1500
PostUp = sysctl -w net.ipv4.ip_forward=1
PostUp = echo 1 > /proc/sys/net/ipv4/ip_forward
PostUp = ufw allow 443/udp || true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PostDown = sysctl -w -q net.ipv4.ip_forward=0
PostDown = echo 0 > /proc/sys/net/ipv4/ip_forward

[Peer]
# infra-1
PublicKey = pub11111111111111111111111111111111111111111
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = xxxxxxxxxx:443

[Peer]
# compute-2
PublicKey = pub2222222222222222222222222222222222222222
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = yyyyyyyyyy:443

[Peer]
# storage-1
PublicKey = pub3333333333333333333333333333333333333333
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = zzzzzzzzzzz:443

10.50.54.2:

[Interface]
# infra-1
Address = 10.50.54.2/24
PrivateKey = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
ListenPort = 443
MTU = 1500
PostUp = sysctl -w net.ipv4.ip_forward=1
PostUp = echo 1 > /proc/sys/net/ipv4/ip_forward
PostUp = ufw allow 51820/udp || true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PostDown = sysctl -w -q net.ipv4.ip_forward=0
PostDown = echo 0 > /proc/sys/net/ipv4/ip_forward

[Peer]
# primary-1
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = xxxxxxxxxxxx:443

[Peer]
# compute-2
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = yyyyyyyyyyy:443

[Peer]
# storage-1
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AllowedIPs = 10.50.54.0/16, 10.42.0.0/16, 10.43.0.0/16
PersistentKeepalive = 15
Endpoint = zzzzzzzzzzzz:443

Do you have any clue? How this should be designed with Wireguard? Why this configuration does not work?

Thanks!

Krzysztofa Krzysztof
  • 141
  • 7

1 Answers1

0

(Apologies, this should probably be a comment as I'm not 100% sure if it's correct, but I don't have enough rep on this site to use comments yet.)

I believe the AllowedIPs for each [Peer] need to be unique (i.e. non-overlapping ranges) because this is how WireGuard knows which peer to route a given outbound packet to. If so, you would need to set up separate IP ranges (for pods and services - nodes shouldn't matter) for each site.

I've never actually tried it with overlapping ranges to see what it would do, and the documentation isn't exactly clear on it, but it seems to imply that this isn't supported. (If it were supported, either it would have to broadcast each packet to all peers whose AllowedIPs included the packet's destination address, or use something like ARP to figure out the routing dynamically.)

I'm not too familiar with k3s, so not sure how its multi-site cluster works - is this basically just multiple separate Kubernetes clusters that are centrally managed?

David Alexander
  • 101