0

My Centos machine has a custom routing table foo_table containing just a default gateway route:

# ip route add default via 10.0.2.1 table foo_table
# ip route show table foo_table  
default via 10.0.2.1 dev bond0.2

A policy rule routes all traffic from 10.0.2.22 using this table:

# ip rule show
0:      from all lookup local 
32765:  from 10.0.2.22 lookup foo_table 
32766:  from all lookup main 
32767:  from all lookup default

When I traceroute from the 10.0.2.22 Centos machine to another machine on the same local subnet, the packet is routed to the router first, then to the server:

# traceroute -In 10.0.2.60
traceroute to 10.0.2.60 (10.0.2.60), 30 hops max, 60 byte packets
 1  10.0.2.1  3.573 ms  3.972 ms  4.219 ms
 2  10.0.2.60  1.461 ms  1.494 ms  1.700 ms

Naturally, I would like local traffic to not unnecessarily involve the router like this.

In the main routing table, there is of course a link scope route for the local subnet:

10.0.2.0/24 dev bond0.2  proto kernel  scope link  src 10.0.2.22

My understanding is that because foo_table is consulted before main (as per ip rule show), the default route via 10.0.2.1 in foo_table is used instead, even though the destination is a link local address.

Does this mean I have to add a link scope route to foo_table as well?

# ip route add 10.0.2.0/24 dev bond0.2 scope link table foo_table
# ip route show table foo_table  
10.0.2.0/24 dev bond0.2  scope link
default via 10.0.2.1 dev bond0.2

It appears to resolve the issue, but is this the right way to do it?

# traceroute -In 10.0.2.60
traceroute to 10.0.2.60 (10.0.2.60), 30 hops max, 60 byte packets
 1  10.0.2.60  0.339 ms  0.355 ms  0.399 ms

Is my understanding of what's happening correct? (I've minimal experience with routing.)

Frode
  • 103
  • 6

2 Answers2

1

Do you know what the purpose of foo_table is? To me, it seems its effect is to route all traffic from the 10.0.2.22 interface (including to the local subnet) via the router, which you say is what you don't want. What's the contents of /etc/iproute2/rt_tables ? You also don't say what the default route is in the main table.

To avoid it, either add the subnet route to foo_table as you suggest, or just delete the rule:

ip rule del from 10.0.2.22 lookup foo_table

So far as I know you don't need to specify scope link to add a subnet route to foo_table, just 

ip route add 10.0.2.0/24 dev bond0.2 table foo_table

since you're specifying the prefix corresponding to the link anyway. Perhaps I'm misunderstanding.

((Incidentally, I suppose the default route in the main table could be moved to the default table for simplicity, but default appears empty by convention, maybe because default route depends on configured interface.))

Cedric Knight
  • 1,098
  • 6
  • 20
  • Many thanks! The servers are on multiple VLANs, so the foo_table rule/table is to ensure that packets on e.g. VLAN "foo" are replied to via the same route (a similar "bar_table" rule/table exists for the other VLAN). In rt_tables there is just "200 foo_table" and "199 bar_table" in addition to the system default ones. – Frode Aug 20 '17 at 15:53
0

For anyone else coming across this, I found a similar question answered on a sibling site: https://unix.stackexchange.com/questions/27850/routing-tables-and-default-routes

In general, you will see two rules (at least) in your routing table on most flavors of *nix.

You'll have a route to your local network (for this example, 10.11.12.0/24):

10.11.12.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

And one identifying your default route.

0.0.0.0 10.11.12.1 0.0.0.0 UG 0 0 0 eth0

So, what this effectively tells your kernel, is:

send any packets going to 10.11.12.0/24 out eth0, directly, without specifically sending them to the default router (flags: U = route is UP)

send any packets to any address other than the local network (0.0.0.0 matches anything) to the gateway (10.11.12.1) (flags: U = route is Up, G = gateway).

If you have more interfaces, or you are multi-homed, or if you have specific network or host routes configured, you may see more routes than that, but this is sort of the minimum that you'd see on a regular basis.

By configuring and bringing up the interface, the kernel will automatically create that local network route.

You don't need to manually add that route.

Frode
  • 103
  • 6