1

I currently have my mesh setup like this: enter image description here

With the wireguard config similar to this on every node:

[Interface]
Address = 10.1.0.1/32
PrivateKey =
ListenPort = 5888

[Peer] # example public node [1-3]
PublicKey =
AllowedIPs = 10.1.0.2/32
Endpoint = X.X.X.X:5888
PersistentKeepalive = 25

[Peer] # example node behind cgnat [4-6]
PublicKey = 
AllowedIPs = 10.1.0.51/32
PersistentKeepalive = 25

This allows me to ping all the green lines in the graph above. But I can't ping any of the reds between the nodes in the CGNAT. How can I?

ATTEMPT 1 (without Node3)

example CGNAT (Node4)

[Interface] # NODE 4
Address = 10.3.0.3/32
PrivateKey = 

[Peer] # NODE 1, 5 & 6
PublicKey = 
AllowedIPs = 10.3.0.1/32,10.3.0.51/32,10.3.0.52/32
Endpoint = X.X.X.X:5888
PersistentKeepalive = 25

[Peer] # NODE 2
PublicKey = 
AllowedIPs = 10.3.0.2/32
Endpoint = X.X.X.Y:5888
PersistentKeepalive = 25

example Public Endpoint (Node1)

[Interface] # NODE 1
Address = 10.3.0.1/32
PrivateKey = 
ListenPort = 5888

[Peer] # NODE 5
PublicKey = 
AllowedIPs = 10.3.0.51/32
PersistentKeepalive = 25

[Peer] # NODE 6
PublicKey = 
AllowedIPs = 10.3.0.52/32
PersistentKeepalive = 25

[Peer] # NODE 2
PublicKey = 
AllowedIPs = 10.3.0.2/32
Endpoint = X.X.X.Y:5888
PersistentKeepalive = 25

[Peer] # NODE 4
PublicKey = 
AllowedIPs = 10.3.0.3/32
PersistentKeepalive = 25

I have also ran on Node1:

$ sysctl -w net.ipv4.ip_forward=1
$ sysctl -w net.ipv4.conf.maxnet.forwarding=1

where maxnet is my wg name.

maxisme
  • 159
  • 1
  • 10

1 Answers1

1

If nodes 4,5,6 are each behind their own CGNAT, then as usual, they can't manage to do proper communication between them. They have to rely on nodes 1,2,3 to relay traffic. So these CGNAT nodes will require extra AllowedIPs for some public peers in their settings to allow other CGNATs nodes through the public nodes, and the public nodes must be set as routers.

Example for node 4 assumed to have IP address 10.1.0.51/32 and using node 2 as routing node:

[Interface]
Address = 10.1.0.51/32
PrivateKey =

[Peer] # example public node 1
PublicKey =
AllowedIPs = 10.1.0.1/32
Endpoint = X.X.X.X:5888
PersistentKeepalive = 25

[Peer] # example public node 2
PublicKey =
AllowedIPs = 10.1.0.2/32,10.1.0.52/32,10.1.0.53/32
Endpoint = Y.Y.Y.Y:5888
PersistentKeepalive = 25

[Peer] # example public node 3
PublicKey =
AllowedIPs = 10.1.0.3/32
Endpoint = Z.Z.Z.Z:5888
PersistentKeepalive = 25

Other CGNAT nodes' IP addresses are expected to be routed through node 2 and should be added to that peer's AllowedIPs. This should also add them automatically to the routing table.

Node 2 must now also be a router, at least on its WireGuard interface which will be both ingress and egress. On Linux this would be done for example with (assuming an interface wg0):

sysctl -w net.ipv4.conf.wg0.forwarding=1

Its firewall rules must also allow forwarded traffic on wg0.

Please note that there's no need to define the other CGNAT peers on the CGNAT nodes, and if defined they must not have other CGNAT nodes' IP addresses in AllowedIPs:

  • they won't be reachable directly and,
  • if the same address is set in subsequent AllowedIPs it will be deleted from each previous definition and only the last defined peer will have it. With cryptokey routing IP address(es) <=> peer.

Nodes 5 and 6 must have a compatible configuration (also using node 2 as router). You could also imagine having instead:

  • split roles where 4 and 5 are routed by node 2, 4 and 6 by node 3 and 5 and 6 by node 1,
  • or having multiple IP addresses within different IP blocks so each block can be routed through a different routing node,
  • or using onion routing with WireGuard traffic within WireGuard traffic so that the routing nodes can't decode traffic not intended for them,
  • or using multiple independent WireGuard interfaces (which aren't subject to cryptokey interaction between themselves, but only to the routing table),
  • or some combination of previous options,

[...]

In all cases, for any topology change even if due to a failure rather than an intended change, a way to synchronize configuration changes in all affected peers is needed and there's currently no dedicated tool to do this.

As a conclusion, here's a blog where BGP (that would be the missing tool) is used along WireGuard, with multiple addresses and also one interface per peer node with only that peer defined to sidestep cryptokey routing. There's probably something to be learnt from this but the topic is way too advanced for me.

Route-based VPN on Linux with WireGuard

A.B
  • 9,037
  • 2
  • 19
  • 37
  • thank you so much for your answer trying it now! – maxisme Dec 07 '20 at 20:34
  • weirdly I now can't ping from any of the nodes behind the CGNATs to the public nodes apart from the router one (i.e nodes 4-6 cant ping 2 & 3) but 2&3 can ping 4-6. – maxisme Dec 07 '20 at 23:26
  • also it appears as if the ordering matters... like you must have all the CGNAT Peers before the router Peer – maxisme Dec 07 '20 at 23:30
  • If the order changes the results, that probably means you have duplicates in AllowedIPs from multiple peers. You must not have any duplicate. For the pings no idea. Maybe Firewall and NAT settings. – A.B Dec 08 '20 at 04:04
  • I insist on the duplicates, I already wrote it in the answer: don't define CGNAT peers at all on CGNATs, and if you insist on defining them (even if they'll be useless), don't set AllowedIPs on them. – A.B Dec 08 '20 at 07:45
  • Hey, Sorry I am still not managing to get this to work. I have put my `attempt1` above but it still can't interact between nodes 4,5,6. Can you see what I have done wrong? – maxisme Jan 04 '21 at 21:40
  • Fixed by adding: – maxisme Jan 05 '21 at 00:37