0

Using NetworkManager, I am trying to setup multiple interfaces on different VLANs; the primary interface, em1, is used on a development VLAN; the secondary interface is on a dedicated iSCSI VLAN.

I am using kernel arguments to launch the host from an iSCSI disk:

GRUB_CMDLINE_LINUX="netroot=iscsi:@172.16.250.250::3260:em2:::iqn.2000-01.com.example.san:disk rd.iscsi.initiator=iqn.2000-01.com.example.node:root rd.lvm.lv=centos/root

To prevent dracut from overriding changes to network config files, I added an override to the dracut config file, the rebuilt:

echo 'omit_dracutmodules+="ifcfg"' >> /etc/dracut.conf

dracut -f
grub2-mkconfig -o /boot/grub2/grub.cfg

After a reboot, I used nmtui to configure the adapters, this built config files with these parameters:

DEVICE=        em1        em2
ONBOOT=        yes        yes
BOOTPROTO=     dhcp       dhcp
DEFROUTE=      yes        no
PEERDNS=                  no
PEERROUTES=               no

I need em1 to be the primary / default interface, em2 is to dedicated to iSCSI; the issue however is that em2 is still adding default routes. After a reboot:

$ ip route show
default via 172.16.250.1 dev em2       <-- WRONG
default via 172.16.100.1 dev em1 proto dhcp metric 100
172.16.100.0/24 dev em1 proto kernel scope link src 172.16.100.100
172.16.100.0/24 dev em1 proto kernel scope link src 172.16.100.100 metric 100
172.16.250.0/24 dev em2 proto kernel scope link src 172.16.250.100
172.16.250.0/24 dev em2 proto kernel scope link src 172.16.250.100 metric 101

There are no route- config files.

Here are some related nmcli options:

$ nmcli c show $if | grep ipv4

                           em1          em2
ipv4.routes                --           --
ipv4.route-metric          -1           -1
ipv4.route-table           0 (unspec)   0 (unspec)
ipv4.routing-rules         --           --
ipv4.ignore-auto-routes    no           yes
ipv4.ignore-auto-dns       no           yes
ipv4.never-default         no           yes

I am not certain what is generating this extra route, how can I check what is creating it and prevent it?

Matt Clark
  • 655
  • 1
  • 8
  • 24
  • Added relevant options. I also tried with `ip=em1:dhcp \ ip=em2:dhcp \ bootdev=em1` before I stopped _dracut_ from touching the config files. – Matt Clark Jul 26 '20 at 23:01
  • The lower metric will override the other route. – Ron Maupin Jul 27 '20 at 00:48
  • My point is that you can set the metric to be worse (higher) than others. The lowest metric wins, but you can change the metrics. – Ron Maupin Jul 27 '20 at 01:32
  • I simply made a comment about how it works. I was not putting in an answer. You are already asking the server guys about how to solve the problem, but I was pointing out that the metric tells you which of equal length routes will be used. I know how to change the metrics on my PC, but I'm not sure about on your server. – Ron Maupin Jul 27 '20 at 01:39
  • Got it, thanks. Would like to just figure out where the route is coming from to begin with :) – Matt Clark Jul 27 '20 at 01:49

1 Answers1

1

It looks like you're getting your IP address via DHCPv4. In this case, by default, the default gateway will be set based on whatever the DHCP server sends. If you use DHCP on two distinct interfaces, as you have done here, you may get two gateways - which in this case conflict.

You can tell NetworkManager to ignore one of the gateways by configuring the connection explicitly as shown here.

For example (substitute your connection's actual name for em2):

nmcli c mod em2 ipv4.never-default yes

This will cause the default gateway from the DHCP server to be ignored. The equivalent GUI option is: "Use this connection only for resources on its network".

In addition, I suspect your mystery extra routes are coming from dracut when it brings up the network interface (with dhcp). You should consider manually configuring the em2 interface on the kernel command line rather than using DHCP.

Finally you should have a long chat with your network people about why there is a default gateway being advertised on your storage network at all.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • I hate to say it, but I had already set this for the interface; no change. I added some of the nmcli options to my post. – Matt Clark Jul 27 '20 at 01:54
  • I am my network people... :D :D :D So that is the root of my issue, is the DHCP lease should not include the default gateway? I'll fix that first, then go static on dracut if it's still an issue. – Matt Clark Jul 27 '20 at 02:32
  • ""Finally you should have a long chat with your network people"" I like to pretend I know what I'm doing, even when I don't! The issue here was that my DHCP server was sending out a default gateway on the iSCSI VLAN. I removed that and things now seem to be working as expected. – Matt Clark Jul 27 '20 at 02:58