Configuring multiple Ethernet connections in a linux server

Question

I have the following topology

                                                                                        .───────────────.        
   ┌──────────────────┐                                                             _.─'                 `──.    
   │                  │                                                          ,─'                         '─. 
   │    ISP Router    │ Dynamic IP                                              ;                               :
   │     with NAT     ├─────────────────────────────────────────────────────────:           Internet            ;
   │                  │                                                          ╲                             ╱ 
   └─────────┬────────┘                                                           '─.                       ,─'  
             │                                                                       `──.               _.─'     
             │  10.0.0.1                                                                 `──────┬──────'         
             │                                                                                  │                
      .──────┴────.                                                                       .─────┴─────.          
   ,─'             '─.                   ┌─────────────────────────┐                   ,─'             '─.       
 ,'                   `.                 │                         │                 ,'                   `.     
;          LAN          :  Static DHCP┌──┴───┐                  ┌──┴───┐ Static IP  ;      Public Net       :    
:                       ;─────────────┤ eth1 │     My Server    │ eth0 ├────────────:                       ;    
 ╲     10.0.0.0/24     ╱     10.0.0.5 └──┬───┘                  └──┬───┘  1.2.3.5    ╲     1.2.3.4/20      ╱     
  `.                 ,'                  │       Debian Buster     │    mydomain.com  `.                 ,'      
    '─.           ,─'                    │                         │                    '─.           ,─'        
       `─────────'                       └─────────────────────────┘                       `─────────'

Because of historical reasons, my ISP uses a separate network to provide static IPs. This means I have to connect my server directly to a special connection that is directly exposed to the internet (in the diagram this is eth0 with IP 1.2.3.5).

However, I use this same server as a NAS in my home and since that's a separate network altogether I connect it using a different interface eth1 and create a static DHCP allocation with address 10.0.0.5. This way, other devices in the home network can find it and access the network mounts.

Lastly, I have a DNS record with my domain, in this example mydomain.com, pointing to 1.2.3.5.

I'm running into a couple of issues:

When I try to access mydomain.com from my LAN it doesn't work. I believe this is because the Server is getting the packet from interface eth0 but then replying through eth1 and the other device is dropping the response since it's from a different IP.
On boot the Server should pick 1.2.3.4 as the network to direct internet traffic to so external access works through mydomain.com. However, since both eth0 and eth1 have internet access through the respective gateways, it sometimes boots and picks 10.0.0.1 as the default gateway and then accessing mydomain.com doesn't work. I believe this is a similar issue with traffic entering through eth0 but leaving through eth1.

While I'm familiar with basic networking stuff, I don't have experience with having a server connected to multiple networks at once like this. Ideally I'd like to configure the server so that:

All internet traffic exits through eth0
All traffic from entering eth0 is replied through eth0
All traffic from entering eth1 is replied through eth1

My server runs Debian, I'd appreciate any help.

EDIT: So after disabling the eth1 gateway and playing a bit with tcpdump I was able to confirm that traffic is entering from eth0 and leaving eth1. The issue is that the ISP is not NATing traffic going to the endpoint 1.2.3.5 and then the server is identifying the 10.0.0.X addresses as located on interface eth1.

This is a routing problem, not a firewall problem. Iptables does basically nothing for you here. — Zoredache, Aug 10 '21 at 17:12
https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html — Zoredache, Aug 10 '21 at 17:13
I see, you are right. I edited the question to reflect that. — Jose Javier Gonzalez Ortiz, Aug 10 '21 at 17:28
Don't use DHCP on your internal network, use static there as well, and do not add a default route. ISP router should have all traffic being NAPT and your server eth0 should see that dynamic IP, never your internal network IPs. Don't "believe" verify with `tcpdump` or similar tools which traffic goes where. — NiKiZe, Aug 10 '21 at 18:07

score 1 · Answer 1 · answered Aug 11 '21 at 15:31

1

This is a routing issue which can be solved with source-based routing or policy routing. This involves two routing tables, one for each source address:

See http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html for more details.

answered Aug 11 '21 at 15:31

Falstone

179
6

I tried this as one of the comments suggested. However, it's not effective since the ISP is not NATing traffic to `1.2.3.5`, so packets from ips `10.0.0.X` are reaching `eth0` and then the routing tables are making them return through `eth1`. – Jose Javier Gonzalez Ortiz Aug 11 '21 at 15:40
I wonder if altering the hosts file on each of the hosts in the 10.0.0.X network so that they send requests to mydomain.com to 10.0.0.5. Or you could set up a local DNS, something like this: https://tech.surveypoint.com/posts/installing-a-local-dns-server-behind-a-hardware-router/ – Falstone Aug 13 '21 at 11:16

score 0 · Answer 2 · answered Aug 11 '21 at 00:27

I believe this is because the Server is getting the packet from interface eth0 but then replying through eth1.

No. The ISP Router with NAT will be NATing your client 10.0.0.0 address to an ISP dynamics address, so your server would be replying to that address rather than the internal one. There are exceptions such as FTP that are not necessarily compatible with NAT as they will pass your internal IP address to the other end of the connection, however for the general case your NAT traffic will stay on eth0.

Check your logs to confirm clients are going outbound to 1.2.3.4 and the server is getting connections from the NATed addresses ip.ip.ip.ip.

On boot the Server should pick 1.2.3.4 as the network to direct internet traffic to so external access works through mydomain.com. However, since both eth0 and eth1 have internet access through the respective gateways

Do not do that. Having multiple default gateways in multi-homed configurations rarely works as people intend. Very roughly multiple default gateways should only be used when both paths provide access to identical networks such as for router redundancy/high availability in an enterprise setting. Since your ISP router to the Internet and your direct connection to the public net are not identical you should remove this. eth0 should have a default route, eth1 should have a route to 10.x.x.x. You can still do this with DHCP, however I would approach this as a reservation/exclusion for 10.0.0.5 and configure the server IP manually.

Aside from the extraneous default route, what you described is the normal default operation. Be sure to look for other issues not described above:

Host files differing from DNS (did you try getting clients to connect to 10.0.0.5 instead of 1.2.3.4? I like to use a technical name internally such as webserver1.internalfqdn.com and a friendly name externally such as www.externalfqdn.com)
Manual route tables not reflected above (did you try to shortcut 10.0.0.x to 1.2.3.4 through 10.0.0.5 as a router? - this can work, but it requires extra configuration)
Routing protocols (e.g. RIP) (really a variation on the above, but if you have RIP or other route publishing enabled you may be publishing bad routes

So after disabling the `eth1` gateway and playing a bit with `tcpdump` I was able to confirm that traffic is entering from `eth0` and leaving `eth1`. The issue is that the ISP is not NATing traffic going to the endpoint `1.2.3.5` and then the server is identifying the `10.0.0.X` addresses as located on interface `eth1`. — Jose Javier Gonzalez Ortiz, Aug 11 '21 at 15:02
If they are not NATing the entire configuration is invalid. The 10.x.x.x/8 subnet must not appear on the Internet as it is a reserved address space. I'd suspect something else is up, as I would never expect even a terrible ISP to do something that poorly. — Doug, Aug 28 '21 at 19:14

score 0 · Accepted Answer · answered Aug 14 '21 at 17:44

I ended up solving the issue by using a split DNS approach, deploying a DNS server on my host and making the whole LAN use that as the default DNS server. There, I added custom DNS rewrite rules for mydomain.com to redirect to 10.0.0.5 instead of the public address. This way, hosts in the LAN resolve my domain to the internal IP and hosts outside the LAN resolve it to the public facing IP, with the benefit of local traffic not having to exit through the ISP network like before.

Another alternative that I believe should be able to solve this problem is hairpin routing. However, the router my ISP provides does not seem to support that.

Configuring multiple Ethernet connections in a linux server

3 Answers3