How do I set up the default route on a Linux router with dual ISP links?

Question

So, here's the situation.

I have dual ISP links currently active and working for routing traffic from my network, with a very similar iptables setup to the accepted answer at Load balancing & NAT-ing multiple ISP connections on Linux

I've got two lines in my /etc/iproute2/rt_tables that look like:

...
10 COMCAST
20 CENTURYLINK
...

I set up the routes in each table with the proper default gateway, and set up the rules like so:

ip rule add fwmark 1 table COMCAST prio 33000
ip rule add fwmark 2 table CENTURYLINK prio 33000

And then I set up iptables-based packet marking and routing:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CONNMARK   all  --  anywhere             anywhere             CONNMARK restore
ACCEPT     all  --  anywhere             anywhere             mark match ! 0x0
MARK       all  --  anywhere             anywhere             MARK set 0x1
MARK       all  --  anywhere             anywhere             statistic mode random probability 0.33300000010 MARK set 0x2
CONNMARK   all  --  anywhere             anywhere             CONNMARK save

The problem is, if I leave the system-wide default route (as in ip route list with no table arguments) in place for either ISP, it seems like that overrides the iptables routing for client machines - no packets go over the non-default connection, according to watch -n 1 ifconfig [interface]. If I delete the system-wide default route, routing works great for all of my client machines, with 33%-ish of the packets going over the CenturyLink line, and the rest over the Comcast line. This is awesome!

However, not having a default route breaks everything that needs an internet connection on the router itself. I can't run apt-get update on the router, for example. So, how do I set up a default route for the system that won't override the iptables routing setup? I attempted to set up the multipath route according to the LARTC site's instructions like so:

ip route add default scope global nexthop via $P1 dev $IF1 weight 1 nexthop dev $IF2 weight 1

(I had to slightly modify from the LARTC guide, because the CenturyLink connection's default route doesn't have a gateway IP, just dev ppp0, and it works for that table's default gateway.)

But the multipath route breaks everything on the server AND the clients that it's routing for. Help?

score 0 · Accepted Answer · edited Apr 13 '17 at 12:14

I was able to figure this out on my own.

The solution, in case anybody else runs into a similar situation, was to alter the priority of the ip rule entries for fwmark. I didn't read lbt's answer to the related question carefully enough; he specifically points out that setting the priority to 33000 will make these rules take place after the default table lookup, which is the opposite of what I want. So, I altered the rules to look like this:

100:    from all fwmark 0x1 lookup COMCAST
100:    from all fwmark 0x2 lookup CENTURYLINK

And now, having a default route doesn't break the firewall-based routing, and I can run apt-get and all of my other programs without any issues. Hooray!

How do I set up the default route on a Linux router with dual ISP links?

1 Answers1