[First post here...]
Hello everyone,
I have a CentOS 7 box which is supposed to become a node of a ganeti-cluster. It has 4 interfaces: eno1, eno2 - on the mainboard ens23s0f0, ens23s0f1 - a dualport 10G Intel adapter
I have renamed the 10G interfaces to eth0/eth1 by adding two lines to /usr/lib/udev/rules.d/60-net.rules like this:
ACTION=="add", SUBSYSTEM=="net"m DRIVERS=="?*", ATTR{type}=="1", ATTR{ADDRESS}=="xx:xx:xx:xx:xx:xx", RESULT=="?*", NAME="eth0"
ACTION=="add", SUBSYSTEM=="net"m DRIVERS=="?*", ATTR{type}=="1", ATTR{ADDRESS}=="xx:xx:xx:xx:xx:xy", RESULT=="?*", NAME="eth1"
3 interfaces are in the same subnet 172.16.66.0/24 - eno1, eth0, eth1. That may not be the optimum setup, but let's just accept it for now, OK? eno2 is not needed.
The patchcables of eno1 and eth1 are plugged into the same switch, default VLAN. eth0 is plugged into another switch with a seperate VLAN for cluster data replication.
What I want is that the 10G-interface eth1 persistently becomes the default gateway for the box. However, as long as eno1 is up, the box will pick it as the default gateway. And if it's down, it will pick eth0.
I have no entries in /etc/sysconfig/network or anywhere else. Also I have no files /etc/sysconfig/network-scripts/route-devN.
The ifcfg-xxxN files are basically identical:
HWADDR=xx:xx:xx:xx:xx:xx
NM_CONTROLLED=no
DEVICE=devN
NAME=devN
TYPE=ETHERNET
BOOTPROTO=static
ONBOOT=yes
IPADDR=172.16.66.nn
PREFIX=24
In ifcfg-eth1 I added
GATEWAY=172.16.66.200
DNS1=172.16.66.200
So this is the only file with a GATEWAY directive, however it seems to be ignored.
I am able to set the default route temporarily with
# ip route add default via 172.16.66.200 dev eth1
but as expected this does not survive a reboot.
Finally my routing table would always look like this:
# ip r
default via 172.16.66.200 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev eth0 scope link metric 1004
169.254.0.0/16 dev eth1 scope link metric 1005
172.16.66.0/24 dev eno1 proto kernel scope link src 172.16.66.n1
172.16.66.0/24 dev eth0 proto kernel scope link src 172.16.66.n2
172.16.66.0/24 dev eth1 proto kernel scope link src 172.16.66.n3
Any help on how to make eth1 the default gateway is really appreciated.
Thanks in advance, Andreas
EDIT: OK, so I decided to start it all from scratch. Here's my log and questions:
I installed CentOS 7.6.1810, Kernel 3.10.0-957. Originally I intended to install 7.2.1511 because this is the version of the other clusternodes, but it's too old for the server-CPU.
I unplugged all patchcables except for the 10G-interface that I want to become the default gateway. In the graphical installer I configured a static IP (172.16.66.11) for that interface. In the dialog there's also a button to set a route. There I set a route for 172.16.66.0, gateway 172.16.66.200 which is the internet router, metric 100.
After the basic installation the internet connection worked. I only installed the MidnightCommander and did not rename any interfaces. I checked the routing table:
# ip r
default via 172.16.66.200 dev enp23S0f1 proto static metric 100
172.16.66.0/24 dev enp23s0f1 proto kernel scope link src 172.16.66.11 metric 100
172.16.66.0/24 via 172.16.66.200 dev enp23s0f1 proto static metric 100
This is what I had expected. After I cleared and edited some stuff the ifcfg-enp23s0f1 looked like this (the quotes were set automatically by the installer):
TYPE="Ethernet"
BOOTPROTO="static"
DEFROUTE="yes"
NAME="enp23s0f1"
DEVICE="enp23s0f1"
ONBOOT="yes"
HWADDR="XX:XX:XX:XX:XX:XX"
IPADDR="172.16.66.11"
PREFIX="24"
GATEWAY="172.16.66.200"
DNS1="172.16.66.200"
I noticed that the installer created a file /etc/sysconfig/network-scripts/route-enp23s0f1. The format however is different from what Jenny suggested and what I also found elsewhere:
ADDRESS0=172.16.66.0
NETMASK0=255.255.255.0
GATEWAY0=172.16.66.200
METRIC0=100
As everything was working fine, I added a second interface now (eno1, mainboard, 1Gb/s). I edited ifcfg-eno1 like this (the installer did not set quotes here, probably because I didn't edit that connection upon installation?):
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=no
NAME=eno1
DEVICE=eno1
ONBOOT=yes
HWADDR=YY:YY:YY:YY:YY:YY
IPADDR=172.16.66.22
PREFIX=24
After restart of the network the routing table looked like this:
# systemctl restart network
# ip r
default via 172.16.66.200 dev enp23S0f1 proto static metric 101
172.16.66.0/24 dev eno1 proto kernel scope link src 172.16.66.22 metric 100
172.16.66.0/24 via 172.16.66.200 dev enp23s0f1 proto static metric 100
172.16.66.0/24 dev enp23s0f1 proto kernel scope link src 172.16.66.11 metric 101
So the system automatically sets the metric for eno1 to 100 although I have assigned this vaue to enp23s0f1 in the route-enp23s0f1 file. The metric for enp23s0f1 is then automatically set to 101. The system seems to insist on giving the mainboard-interfaces priority over the PCIe-adapter.
The above happened with NetworkManager still being active. I deactivated NM then
# systemctl stop NetworkManager
# systemctl disable NetworkManager
and added to ifcfg-enp23s0f1 and -eno1 as first line
NM_CONTROLLED=no
The routing table now looks like this:
# systemctl restart network
# ip r
default via 172.16.66.200 dev eno1
169.254.0.0/16 dev eno1 scope link metric 1002
169.254.0.0/16 dev enp23s0f1 scope link metric 1005
172.16.66.0/24 dev eno1 proto kernel scope link src 172.16.66.22
172.16.66.0/24 dev enp23s0f1 proto kernel scope link src 172.16.66.11
Again eno1 is set as default route. The file route-enp23s0f1 is obviously being ignored and only seems to be relevant with NetworkManager enabled.
Do I need to enable NetworkManager at all, is that advisable? I thought not.
Why does the system automatically change the metric in favor of the mainboard-if even with NM enabled and an explicit value given in the route-file?
Thanks and best regards Andreas
