Linux adds the wrong default route

Question

For some reason my Centos 5.9 Linux 2.6.18 x86_64 wants me to have two default routes.

Whenever I reboot the server, my routing table looks like this:

[root@server1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
255.255.255.255 0.0.0.0         255.255.255.255 UH    0      0        0 bond0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 bond1
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 bond0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 bond1
0.0.0.0         192.168.10.1    0.0.0.0         UG    0      0        0 bond0
0.0.0.0         192.168.0.254   0.0.0.0         UG    0      0        0 bond1

bond1 is a local network, so having a last default route pointing to this network makes all internet requests fail.

It is easily fixed by executing route del default gw 192.168.0.254 and I could possibly add that command to some startup script. I would however like to understand what's going on and get to the root of the problem.

I hope anyone can tell me, why this happens. My research confirm that there should always be only one default gateway, but I can find no answer to why there would automatically be two of them.

Here is some config files:

[root@server1 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=myhostname.com

[root@server1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.10.11
NETMASK=255.255.255.0
NETWORK=192.168.10.0
BROADCAST=192.168.10.255
GATEWAY=192.168.10.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

[root@server1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
IPADDR=192.168.0.15
NETMASK=255.255.255.0
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
GATEWAY=192.168.0.254
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

I understand that swapping bond0 and bond1 would make the two default routes switch place as well, effectively giving me internet access on boot. But I still think it's not the good solution.

Around the internet people are talking about files at /etc/sysconfig/network-scripts/route-X, I don't have any of those though.

Thanks for your time.

Answer 1

The "default" routes are set based on the GATEWAY lines in your ifcfg-<interface> files. As Dom mentioned in his comment, if you remove the incorrect GATEWAY line, your routing table will be as you expect.

Answer 2

There's a difference between GW and Default GW. Based on your config, it looks like you want both interfaces to be able to access external networks. You could remove the GATEWAY line as has been recommended but doing so will prevent bond1 from any external communication.

As soon as you go multihomed I find it's a good idea to use policy based routing. Also known as split-access routing. It's a good habit to get into and is required when you want to put multiple interfaces on the same subnet.

For your case specifically I would:

• Edit /etc/iproute2/rt_tables
• At the bottom of the file add:
  • 100 bond0tbl
  • 101 bond1tbl

Now you have defined your internal tables called bond0tbl and bond1tbl respectively. Now you need to create the rules for these tables

• Open/Create /etc/sysconfig/network-scripts/rule-bond0
• Add the following: from 192.168.10.11 table bond0tbl
• Open/Create /etc/sysconfig/network-scripts/rule-bond1
• Add the following: from 192.168.0.15 table bond1tbl

Now that you've defined the rules, it's time to define the routes.

• Open/Create /etc/sysconfig/network-scripts/route-bond0
• Add the following:
  • default via 192.168.10.1 dev bond0 table bond0tbl
  • 192.168.10.0/24 via 192.168.10.11 dev bond0 table bond0tbl
  • 192.168.10.0/24 via 192.168.10.11 dev bond0 table main
• Open/Create /etc/sysconfig/network-scripts/rule-bond1
• Add the following:
  • default via 192.168.0.254 dev bond1 table bond1tbl
  • 192.168.0.0/24 via 192.168.0.15 dev bond1 table bond1tbl
  • 192.168.0.0/24 via 192.168.0.15 dev bond1 table main

Last I would remove the GATEWAY line from BOTH ifcfg-devX files and add it to /etc/syconfig/network.

It's easy to get confused about the GW that a specific interface should use for routing and the Default GW that EVERYONE should use...in the event that the desired route doesn't exist. Setting it in /etc/sysconfig/network has always felt more "global" to me.

When your ducks are all in a row you service network restart or ifup/ifdown or reboot to get everything created. To see if it worked you can do:

• ip route show table bond0tbl
• ip route show table bond1tbl
• ip route show table main
• ip rule show

To recap,

1. Define tables with a meaningful naming convention
2. Define rules to force traffic for a specific IP to a specific interface
3. Define the default route for that interface/IP
4. Define the route to be added to the newly created table
5. Define the route to be added to the "main" table.
6. Let the ifup/service network restart/reboot do the work

Answer 3

Put in these line in your /etc/sysconfig/network file if you want bond0 as your default gateway. If you want bond1 as your default gateway the put GATEWAYDEV=bond1.

NETWORKING=yes
HOSTNAME=mydbserver
GATEWAYDEV=bond0
GATEWAY=192.168.xxx.xxx