1
We have a growing robotics setup for which we need to improve the network configuration. The fact that is is a robotic setup is not important for the question itself, but it could IMO help understanding our needs.
The network scheme is as follows:
So we have a few robots, each of which can be connected via wifi to a common network, and also runs its own internal network. What we started to need now is to access computers in the internal networks on the robots from outside (over the wifi link). The scheme shows the current state of our setup. All PCs are running either Ubuntu or Raspbian.
The main thing that's running on the robots is ROS. It's a pub-sub message passing system. Each robot runs a "server"/broker on the main PC. All publishers get a random free port from the system they're running on (this can't be configured) and pass it to the broker in form "topic_name:hostname:port". Subscribers then contact the broker, get the address of publishers of topics of interest, and then create direct P2P connections with the publishers (again, using the "hostname:port" info they got from the broker).
What we'd like to keep is having a unified setup on the robots (i.e. network settings etc.). We often need to swap sensors or computers from one robot to another, and having to change lots of configs with each of these changes would not be acceptable. Also, most of the colleauges working with the robots are not networking experts, so there's need to keep things as simple (close to defaults) as possible.
The basic requirement for the network is that the robots can function completely isolated (i.e. without the wifi link which provides DHCP server). That's why we use static IPs in the internal network. But as soon as the wifi link gets available, we'd like to be able to reach x-robot, x-robot-nuc and x-robot-jetson via the wifi link from the ROS subscribers. And we'd also like to allow the internal network to access internet via the wifi link.
What we tried:
- on x-robot:
iptables -t nat -A POSTROUTING -m iprange --src-range 192.168.1.98-192.168.1.98 ! --dst-range 192.168.2.0-192.168.2.255 -o eth_bullet -j MASQUERADE- this allows x-robot-jetson to access internet, but nothing else - on x-robot:
iptables -t nat -A POSTROUTING -o eth_bullet -j MASQUERADE- this allows x-robot-jetson to access internet and other computers in the network, but then packets for e.g. sensor 192.168.1.72 from y-robot start to "leak" to x-robot, too, and routing gets crazy - on x-robot: bridging eth_laser and eth_bullet to bridge br0. On the additional computers (x-robot-jetson etc.), set a connection that uses both DHCP client and static address 192.168.1.* for use in the internal network. This is closest to what I'd want, but again, packets for 192.168.1.72 on x-robot start to "leak" to y-robot, and routing gets crazy, too... The idea here is that communication that should stay internal to the robot will use the 1.* network, while communication with "outside" will use the 2.* network that got there via DHCP over the bridge.
- NAT on x-robot is not an option since the subscribers require a working connection from outside to
x-robot-jetson:any_port.
So, assuming that the 3rd option with bridging eth_bullet and eth_laser is not a completely bad idea, how would I set it so that DHCP and 192.168.2.* network flow over the bridge, but 192.168.1.* (and possibly other subnets) stay isolated from each other? I played around with arptables, ebtables, iptables and such, but I never found a solution that would prevent ARP requests to leak to other robots. Then routing goes mad and bad things start to happen...
Martin Pecka
Posted 2020-01-21T12:17:19.417
Reputation: 643
I don't understand the part about allowing "192.168.2.* network flow over the bridge". That doesn't match what you described earlier: if the subscriber PCs have "a connection that uses both DHCP client and static address 192.168.1.* for use in the internal network", then ... the 192.168.2.x network isn't used at all when they're trying to reach the robots, so it's not going over the bridge anyway – the 192.168.1.x network is used instead. – user1686 – 2020-01-21T12:30:29.717
1Is there anything stopping you from segmenting your network further? So you'd use 192.168.1 for management, 192.168.2 for robot Y and 192.168.3 for robot Z? In that case by setting up possibly static routes (or blocks) you could make sure which traffic is allow to go where or are those sensor addresses tied to the sensors instead of the robot? – Seth – 2020-01-21T12:32:09.780
@user1686: Ok, I didn't state that pretty well. "Additional computers" in the bridging attempt are x-robot-jetson etc. While what we need is to access these machines from subscribers which are behind the wifi link, thus have only the 192.168.2.* address from DHCP. – Martin Pecka – 2020-01-21T12:35:08.373
1@Seth: Segmenting the network would solve the problem. But it doesn't allow for having a unified networking configuration on all robots (since each robot would need to have a different network for internal communication). If we won't find another solution, we'll go this way, but I feel that there should be a solution even for the case of identical internal networks. And yes, the sensor addresses are tied to the sensor, not the computer they're attached to. And it's a pain to reconfigure IP addresses in some sensors. – Martin Pecka – 2020-01-21T12:38:23.627
Right, okay then. I'm curious about case #2 now: the NAT rule on its own shouldn't cause any extra leakage, and especially not to a subnet two hops over; do you have any packet captures of that happening? – user1686 – 2020-01-21T14:00:47.487
@user1686 Unfortunately I don't have the packet captures. It's possible that they weren't IP packets leaking, but just the ARP ones, which broke routing. – Martin Pecka – 2020-01-21T14:12:14.937
Well, ARP is even more difficult to "leak" through a router given that it's link-local. Overall the NAT option should be fine for outgoing-only access (that's pretty much how ISPs handle hundreds of 192.168.1.x customer LANs, isn't it?) So this needs more thorough investigation, IMHO. – user1686 – 2020-01-21T14:53:45.467