tl;dr: How can i get 3 NIC to route, backup/failover and communicate together?
I got a proxmox server with 3 NIC vmbr1: company lan - dhcp (engineering department) vmbr2: robotics network (static) vmbr3: company lan - different subnet (business department)
Now, i'd like to offer VM over vmbr1 and vmbr3. vmbr2 has access to some robots and automation systems.
I ssh via engineering subnet to vmbr1 and have access to internet via wpa_supplicant (working only on vmbr1). Sometimes i lose connection after restart/reload due to the bad routes and not getting dhcp offer on all.
vmbr1 and vmbr3 is bridged to my vms and port forwards to my pfsense which serves dhcp in robonet (vmbr2) My wish would be to have mainly internet over vmbr1, and serve the vms over both interfaces.
I barely grasp the routing idea but can't see the final touch to get it running.
interfaces:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface eno2 inet manual
auto eno1
iface eno1 inet static
hwaddress ac:1f:6b:e9:6e:56
# zuweisung virtueller FAKE MAC um doppelte dhcp anfrage zu vermeiden
auto vmbr1
iface vmbr1 inet dhcp
hwaddress ac:1f:6b:e9:7e:56
wpa-conf /etc/wpa_supplicant/wpasupp.conf
wpa-driver wired
wpa-logfile /var/log/eit-wpa.log
wpa-debug-level 2
#pre-up /etc/network/preup.sh
#post-down /etc/network/postdown.sh
bridge-ports eno1
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A PREROUTING -i vmbr1 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 192.0.0.2
post-up iptables -t nat -A PREROUTING -i vmbr1 -p udp -j DNAT --to 192.0.0.2
post-up ip route replace default via 10.152.32.254 dev vmbr1
# eit-w1, Zuweisung der originalen NIC ENO1 MAC.
# Öffnen der Firewall für alle ports außer 22 und 8006. April2022: 80 und 443 für nginx baremetal - wurde wieder entfernt für tests (16.8.2022)
# Öffnen aller udp ports zur pfsense
auto enx00e04c680542
iface enx00e04c680542 inet manual
altname eno3
# USB NIC für vmbr3 IRAS-W0
auto vmbr3
iface vmbr3 inet dhcp
hwaddress ac:1f:6b:e9:7e:73
bridge-ports enx00e04c680542
bridge-stp off
bridge-fd 0
#post-up route del default dev vmbr3
#post-up ip route add 10.181.116.0/22 dev vmbr3 src 10.181.116.185 table irasnet
#post-up ip route add default via 10.181.116.254 dev vmbr3 table irasnet
#post-up ip rule add to 10.181.116.185/32 table irasnet
#post-up ip rule add from 10.181.116.185/32 table irasnet
# IRAS W0 Brücke für VM ans ADS - Netzwerk. Funktioniert auch...
auto vmbr2
iface vmbr2 inet static
address 172.31.1.25/16
bridge-ports eno2
bridge-stp off
bridge-fd 0
hwaddress ac:1f:6b:e9:7f:60
# robonet, lokaler Zugriff über eno2. NAT Forwarding kommt dann über die Pfsense
auto vmbr100
iface vmbr100 inet static
address 192.0.0.1/30
bridge-ports none
bridge-stp off
bridge-fd 0
hwaddress ac:1f:6b:e9:7f:61
post-up iptables -t nat -A POSTROUTING -s '192.0.0.0/30' -o vmbr1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.0.0.0/30' -o vmbr1 -j MASQUERADE
# firenet Brücke zur Firewall und nur für Kommunikation von Server zu Firewall mit NAT und PAT
#https://linuxhint.com/debian_network_interface_setup/
#https://wiki.debian.org/NetworkConfiguration#Using_DHCP_to_automatically_configure_the_interface
Update:
Maybe my base idea is wrong - iam not sure if i should configure it via iptables and prerouting or connect the pfsense directly to campusnet (companylan). Pic1 is the goal to put out different web services and reverse proxy them with pfsense+haproxy or nginx which are hosted on this very server (myproxmox)
- myproxmox.campus.net/app1... (e.g. pfsense, siemens hmi)
- myproxmox.campus.net/app2...
default via 10.152.47.254 dev vmbr1
10.152.32.0/20 dev vmbr1 proto kernel scope link src 10.152.32.76
10.181.116.0/22 dev vmbr3 proto kernel scope link src 10.181.116.185
172.31.0.0/16 dev vmbr2 proto kernel scope link src 172.31.1.25
192.0.0.0/30 dev vmbr100 proto kernel scope link src 192.0.0.1
