How to correctly set up routing on machine with 4 interfaces so that three of interfaces are on the same subnet?

Question

So I have a linux machine with 4 interfaces:

1. enp1s0
2. enp2s0
3. enp3s0
4. enp4s0

What I would to do is have enp1s0 be the WAN interface and acquire its ip address, dns and gateway via DHCP.

For the other three interfaces, I would like them to have:

• IP addresses on the same subnet
• Have all hosts on one LAN interface be able to see all the other hosts on other LAN interface
• Run shorewall as a firewall
• Use DNS Masq

Here is what I've gotten so far:

cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# listed as #1 
# this is the wan interface
auto enp1s0
iface enp1s0 inet dhcp

# listed as #2
# this routes to upstairs
auto enp3s0
iface enp3s0 inet static
    address 192.168.47.254
    netmask 255.255.255.0
    network 192.168.47.0    
    broadcast 192.168.47.255    

# listed as #4
auto enp2s0
iface enp2s0 inet static
    address 192.168.47.253
    netmask 255.255.255.0
    network 192.168.47.0    
    broadcast 192.168.47.255    

# listed as #3
auto enp4s0
iface enp4s0 inet static
    address 192.168.47.252
    netmask 255.255.255.0
    network 192.168.47.0    
    broadcast 192.168.47.255    

cat /etc/dnsmasq.conf
interface=enp2s0
interface=enp3s0
interface=enp4s0
listen-address=192.168.47.254 # Explicitly specify the address to listen on  
bind-interfaces      # Bind to the interface to make sure we aren't sending things elsewhere  
server=8.8.8.8       # Forward DNS requests to Google DNS  
domain-needed        # Don't forward short names  
bogus-priv           # Never forward addresses in the non-routed address spaces.  
dhcp-range=192.168.47.100,192.168.47.250,12h # Assign IP addresses between 192.168.46.100-250 with a 12 hour lease time 

log-dhcp
log-queries

interface=enp2s0      # Use interface wlan0  
interface=enp3s0      # Use interface wlan0  
interface=enp4s0      # Use interface wlan0  
listen-address=192.168.47.254 # Explicitly specify the address to listen on  
bind-interfaces      # Bind to the interface to make sure we aren't sending things elsewhere  
server=8.8.8.8       # Forward DNS requests to Google DNS  
domain-needed        # Don't forward short names  
bogus-priv           # Never forward addresses in the non-routed address spaces.  
dhcp-range=192.168.47.100,192.168.47.250,12h # Assign IP addresses between 192.168.46.100-250 with a 12 hour lease time 

log-dhcp
log-queries

And here is a sample ip route table from one of the client machines:

ip -one addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0    inet 192.168.46.5/24 brd 192.168.46.255 scope global eth0\       valid_lft forever preferred_lft forever
2: eth0    inet6 fe80::82ee:73ff:fe5d:89d1/64 scope link \       valid_lft forever preferred_lft forever
3: eth1    inet 192.168.47.244/24 brd 192.168.47.255 scope global eth1\       valid_lft forever preferred_lft forever
3: eth1    inet6 fe80::82ee:73ff:fe5d:89d0/64 scope link \       valid_lft forever preferred_lft forever

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.46.1    0.0.0.0         UG    0      0        0 eth0
192.168.46.0    0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.47.0    0.0.0.0         255.255.255.0   U     1      0        0 eth1

What works so far:

• Machines on different interfaces get IP address that are correct
• Machines can either ping all of the interfaces on the router or they can't. I think this might depend on which machine most recently DHCP'ed in
• If I can ping from a client machine, other things work too e.g. ssh
• I've never gotten two client machines to ping each other

After a lot of reading, I've determined that I need either static routes, different subnets for each lan interface or bridging.

Ideally, I would like to be set permissions in shorewall based on the interface and still use one subnet for all the LAN interfaces.

Answer 1

You're on the wrong track altogether, and it doesn't have anything to do with Shorewall (or really any other firewall package) but rather IP networking in general.

If you imagine that a given IP host has three interfaces in the same subnet and a packet arrives for that subnet from the fourth, how does that host know which of the three interfaces should receive the packet? Does it ARP in all three? Does it just send all the traffic to the first one that came up, thus preventing traffic bound for the other two? This is why if you tried to configure this from a shell you'd be given an error message about overlapping subnets.

So - you have three reasonable options, with a couple of sub-options.

1. Dump the other two interfaces and just hook up a single port to a switch and then hang the hosts (and possibly wifi in the same subnet) off of that switch. It's possible to join all three of your internal interfaces into a single bond or team but this basically takes all three physical interfaces and turns them into a single logical interface.

2. Create a Linux bridge interface. Give the IP, DHCP and other configuration elements to the bridge and then tie the three physical interfaces in as members of the bridge. Your firewall sees the single bridge interface as the inside (...even though it ties to three segments). Again - from an IP perspective you have one interface in the inside subnet, not three.

3. Put the three interfaces in three non-overlapping IP subnets and configure accordingly.