2
I have 2 servers each of them has 3 network interfaces. 1 interface ("eno") is used for internet connection via gateway and 2 interfaces ("ens0", "ens1") are used to connect both servers directly point to point. It looks like:
|
gw
| |
S1 == S2
I need to send data from "ens0" to "ens0" and from "ens1" to "ens1" independently. When I'm trying to do this on S2 server I found that all traffic goes through ens0 despite I bind 2 sockets (C++) to different interfaces. When I do this on S1 it works good. So I checked "ip tables". They are:
S1 (works correctly):
default via 192.168.1.4 dev eno1
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.3
192.168.2.0/24 dev ens3f0 proto kernel scope link src 192.168.2.1
192.168.2.0/24 dev ens3f1 proto kernel scope link src 192.168.2.2
S2 (works wrong):
default via 192.168.1.4 dev eno2 proto static metric 100
default via 192.168.1.4 dev ens3f0 proto static metric 101
default via 192.168.1.4 dev ens3f1 proto static metric 102
192.168.1.0/24 dev eno2 proto kernel scope link src 192.168.1.2 metric 100
192.168.1.4 dev ens3f0 proto static scope link metric 100
192.168.1.4 dev ens3f1 proto static scope link metric 101
192.168.2.0/24 dev ens3f0 proto kernel scope link src 192.168.2.3 metric 100
192.168.2.0/24 dev ens3f1 proto kernel scope link src 192.168.2.4 metric 101
Table for S2 was generated by OS.
To fix the trouble I tried to change S2 ip table as for S1. I changed it to the state:
default via 192.168.1.4 dev eno2 proto static
192.168.1.0/24 dev eno2 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev ens3f0 proto kernel scope link src 192.168.2.3
Then when I'm trying to make
sudo ip route add 192.168.2.0/24 dev ens3f1 src 192.168.2.4
I got the error
RTNETLINK answers: File exists
net.ipv4.ip_forward = 0 for both servers.
My questions are:
1) Why I can't add last entity on S2 when it is on S1?
2) How to configure ip tables correctly for both interfaces could operate independently and traffic forwarding for them was impossible? (I want to be sure if I send traffic to specific interface - it will never goes through other one).
floatJoe
Posted 2018-02-22T14:57:05.323
Reputation: 21
Any reason you can't have 192.168.2.0/24 on
ens3f0on both servers, and 192.168.3.0/24 onens3f1on both servers? Then you can choose which interface to use with the destination address. In general, trying to deal with two identical broadcast domains on different network interfaces is asking for a headache. In case this is a XY-Question: What are you trying to achieve by using two connections between the servers? – dirkt – 2018-02-22T15:34:24.243@dirkt Thanks for the advise. It looks as "dirty hack" ) but I'll go this way if I fail my own. The reason I use direct connection is I'm working on parallel distributed calculations with throughput of tens Gbps and hence I don't want to have any additional hw between servers. – floatJoe – 2018-02-23T08:10:38.527
It's a general principle that each interface should be on a distinct broadcast domain, so it's not a "dirty hack", on the contrary. If you are using two interfaces instead of one in order to speed up communication between the servers, consider bonding the interfaces: You'll get double the throughput with just on IP address on each side, without multiple IP address or having to bind applications, or other headaches. – dirkt – 2018-02-23T10:47:31.720
If bonding works out for you, you can answer your own question, describe what you've done, and accept the answer. – dirkt – 2018-02-23T11:39:15.407
I solved the trouble. What I did. I created files /etc/sysconfig/network-scripts/route-ens3f0 and /etc/sysconfig/network-scripts/route-ens3f1 on both servers with routes like "192.168.2.3/32 via 192.168.2.1 dev ens3f0" and so on respectively for each server and each interface. That's all. Everything works fine exactly I wanted to. PS: I didn't experiment with bonding because of first I'm sure it will work and second I wanted to solve problem with 2 interfaces but don't reduce it to 1 interface. @dirkt Great thanks, you pushed me to right way. – floatJoe – 2018-02-26T11:00:29.020