Webserver Multiple Gateways

Question

I have Ubuntu Server 12.04

I have 3 gateways on my lan

10.0.0.1
10.0.0.2
10.0.0.3

Ubuntu has 3 NICs

eth0 10.0.0.60 gw 10.0.0.1
eth1 10.0.0.61 gw 10.0.0.2
eth2 10.0.0.62 gw 10.0.0.3

If a request comes in from gw 10.0.0.1 it work fine. the other gateways don't

I have tried to fix this with routes but I can't seem to get it right.

My ISP can only give 10Mbps up per modem the server streams a live video feed. I need all 3 modems for load balancing

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.3        0.0.0.0         UG    100    0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth2
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

If this could be done with one NIC I am fine with that I just need to be able to balance this traffic.

I tried this (but all networking stopped after):

#echo "60 out60" >> /etc/iproute2/rt_tables
#echo "61 out61" >> /etc/iproute2/rt_tables
#echo "62 out62" >> /etc/iproute2/rt_tables

#ip route add default via 10.0.0.1 table out60
#ip route add default via 10.0.0.2 table out61
#ip route add default via 10.0.0.3 table out62

#ip rule add from 10.0.0.60 table out60
#ip rule add from 10.0.0.61 table out61
#ip rule add from 10.0.0.62 table out62

This Diagram doesn't have the server in question its just so you can see the network layout.

Network Diagram
http://23.31.204.85/netdiag.png

My ISP can only give 10Mbps up per modem the server streams a live video feed. I need all 3 modems for load balancing. — Eric Fluharty, Aug 09 '13 at 02:50
What does you routing table look like? You might have to configure each one to be on a separate subnet. — Ryan Gibbons, Aug 09 '13 at 04:14
This network is in a production environment. I can't change the design of the network, only the server its self. — Eric Fluharty, Aug 09 '13 at 06:05
Could you change gw ip addresses (10.0.0.1/10.0.0.2/10.0.0.3) on the modems? — ALex_hha, Aug 09 '13 at 08:31
No, you don't need 3 modems. You need to host this in a datacenter and get a proper connection. — Dennis Kaarsemaker, Aug 09 '13 at 21:13
A datacenter is in the future just maxing out the resources I have first. — Eric Fluharty, Aug 10 '13 at 00:08

score 0 · Answer 1 · answered Aug 09 '13 at 19:47

0

I will strongly recommend to change subnet. It's very bad idea to use the same subnet for all interfaces and their gateways, imho

I would suggest to use something like the following

eth0 10.0.1.61 gw 10.0.1.1
eth1 10.0.2.62 gw 10.0.2.1
eth2 10.0.3.63 gw 10.0.3.1

You could try to setup PBR, but I don't know, if it would work in your environment

#!/bin/sh

IP1='10.0.0.60'
IF1='eth0'
GW1='10.0.0.1'
P1_NET='10.0.0.0/24'

IP2='10.0.0.61'
IF2='eth1'
GW2='10.0.0.2'
P2_NET='10.0.0.0/24'

IP3='10.0.0.62'
IF3='eth2'
GW3='10.0.0.3'
P3_NET='10.0.0.0/24'

/sbin/ip route add $P1_NET dev $IF1 src $IP1 table ISP1
/sbin/ip route add default via $GW1 table ISP1

/sbin/ip route add $P2_NET dev $IF2 src $IP2 table ISP2
/sbin/ip route add default via $GW2 table ISP2

/sbin/ip route add $P3_NET dev $IF3 src $IP3 table ISP3
/sbin/ip route add default via $GW3 table ISP3

/sbin/ip rule add from $IP1 table ISP1
/sbin/ip rule add from $IP2 table ISP2
/sbin/ip rule add from $IP3 table ISP3

/sbin/ip route add default via $P1
/sbin/ip route flush cache

Don't forget to enable ip forwarding and disable Reverse Path Filtering

# echo 1 > /proc/sys/net/ipv4/ip_forward
# echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter

answered Aug 09 '13 at 19:47

ALex_hha

7,025
1
23
39

I can not modify the network. I have many other production systems running on this network. – Eric Fluharty Aug 09 '13 at 23:40
I will try this again, but last I tried it broke networking. – Eric Fluharty Aug 09 '13 at 23:54
$ sudo ip route add 10.0.0.0/24 dev eth0 src 10.0.0.60 table ISP1 RTNETLINK answers: Invalid argument – Eric Fluharty Aug 10 '13 at 01:03
I removed the scr field from this command, it ran successfully but didn't work in the end `# ip route add 10.0.0.0/24 dev eth2 table ISP3` – Eric Fluharty Aug 10 '13 at 01:23
Did you add ISP1/ISP2/ISP3 to the /etc/iproute2/rt_tables ? – ALex_hha Aug 10 '13 at 06:36
yes i added `1 ISP1 2 ISP2 3 ISP3` – Eric Fluharty Aug 10 '13 at 14:31
According to the Network Diagram that you have provided, I don't understand, where is web server himself? And 3 servers at the bottom of the diagram are they 3 different physical servers? – ALex_hha Aug 10 '13 at 17:19
The server is actually a virtual server on one of those physical servers. – Eric Fluharty Aug 10 '13 at 18:36

inetplumber · Answer 2 · 2013-08-09T03:43:29.167

The confusion expressed in the comments stems from this Private Class A LAN networking scheme being not very conventional or common in practice. Usually each net block is (a /28,/16 etc.) is broken up and the gateway set to the first usable IP. In this case it looks like each modem is a /31.

Paste output of route -n or netstat -nra (should work on most *nix variants) if you're interested as to what gateway each interface believes it has, for further diagnosis.

Webserver Multiple Gateways

2 Answers2