Trying to setup Multi-Homing under Ubuntu12 but can only get 1 of 2 IP to accept requests

Question

I am having an issue on some new servers that we are creating under Ubuntu 12 in a virtual box running under VMware(VSphere I believe). This is at an ISP not my desktop. I think that we have the same issue our old datacenter but never really found a way to fix it. We just stopped using multi-homing and just used one or the other of the IP addresses (we wanted one one in a DMZ and one internal only). We want to have the database only available on the INT subnet

The problem(short version): Trying to config Muli-homing under Ubuntu, but can only get one IP to respond.

Longer version - We want to have two networks, one that is DMZ and can have access to the outside internet and INT which is internal and can only access other servers in our piece of the datacenter. I can define them such that they wil show up correctly in ifconfig, netstat -nr, and IP route show, but if I try and access a service on the server I can only get one ior the other to respond, the other times out. Which one responds stays consistent between reboots.

From everything I've been reading we have things defined correctly, but maybe we are missing something or there is something special that has to also be done since we are under VMware.

If I try SSH or PING'ing into the DMZ address it works fine. If I try the INT address it just times out. There is no indication of it being blocked by iptables, or any other related error messages in the log(s).

If I try each individually they will work. It's only when I try as multi-homing that access fails

Background:

Ubuntu 12.04 LTS under VMware

Names have been changed but have been keep consistent. A and B are different numbers and are replaced consistently. The others are the sameas they are in our config.

Local server name mapping
DMZ_001     10.A.B.64/26
INT_002     10.A.B.128/27

DMZ - 10.A.B.71 (eth0)
INT - 10.A.B.140 (eth1)

Below I have ifconfig, netstat -nr, ip route show and the interfaces file. If there is other info that can help let me know.

=====================

username@servername:~$ cat /etc/network/interfaces
iface lo inet loopback
auto lo

#DMZ
auto eth0
iface eth0 inet static
address 10.A.B.71
netmask 255.255.255.192
gateway 10.A.B.65
#up route add default gw 10.A.B.65
dns-nameservers 8.8.8.8 8.8.4.4

# INT
auto eth1
iface eth1 inet static
  address 10.A.B.140
  netmask 255.255.255.224
  gateway 10.A.B.129
#up route add default gw 10.A.B.129
  dns-nameservers 8.8.8.8 8.8.4.4
  dns-search thinaire.net

================================

username@servername:~$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.A.B.65    0.0.0.0         UG        0 0          0 eth0
10.A.B.64    0.0.0.0         255.255.255.192 U         0 0          0 eth0
10.A.B.128   0.0.0.0         255.255.255.224 U         0 0          0 eth1

================================

username@servername:~$ ip route show
default via 10.A.B.65 dev eth0  metric 100
10.A.B.64/26 dev eth0  proto kernel  scope link  src 10.A.B.71
10.A.B.128/27 dev eth1  proto kernel  scope link  src 10.A.B.140
username@servername:~$

============================

username@servername:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:ac:5e:1b
          inet addr:10.A.B.71  Bcast:10.A.B.127  Mask:255.255.255.192
          inet6 addr: fe80::250:56ff:feac:5e1b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26089 errors:0 dropped:9 overruns:0 frame:0
          TX packets:18282 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3784760 (3.7 MB)  TX bytes:2359060 (2.3 MB)

eth1      Link encap:Ethernet  HWaddr 00:50:56:ac:06:44
          inet addr:10.A.B.140  Bcast:10.A.B.159  Mask:255.255.255.224
          inet6 addr: fe80::250:56ff:feac:644/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4437 errors:0 dropped:8 overruns:0 frame:0
          TX packets:726 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:295739 (295.7 KB)  TX bytes:74047 (74.0 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

score 1 · Accepted Answer · edited Jun 11 '20 at 10:02

1

You can not have more than one default gateway per route table in linux configured this way.

To have multiple default gateways you will need to configure separate route tables for each source IP.

Something like that:

ip route add default via 10.A.B.65 src 10.A.B.71 table 1

ip route add 10.A.B.64/26 dev eth0 table 1

ip route add 10.A.B.128/27 dev eth1 table 1

ip rule add from 10.A.B.71 lookup 1

and for the second interface

ip route add default via 10.A.B.129 src 10.A.B.140 table 2

ip route add 10.A.B.64/26 dev eth0 table 2

ip route add 10.A.B.128/27 dev eth1 table 2

ip rule add from 10.A.B.140 lookup 2

You also may want to add other routes to these route tables

To configure it in /etc/network/interfaces you can put these commands in a script and call in up directive (see man 5 interfaces)

auto eth0 iface eth0 inet static address 10.A.B.71 netmask 255.255.255.192 gateway 10.A.B.65 up /usr/local/bin/eth0-up.sh dns-nameservers 8.8.8.8 8.8.4.4

where eth0-up.sh is a file with:

#!/bin/bash

ip route add default via 10.A.B.65 src 10.A.B.71 table 1

ip route add 10.A.B.64/26 dev eth0 table 1

ip route add 10.A.B.128/27 dev eth1 table 1

ip rule add from 10.A.B.71 lookup 1

Repeat the same for eth1

edited Jun 11 '20 at 10:02

Community

1

answered Jul 22 '14 at 04:02

stimur

894
5
11

Thanks, I'll investigate that today. Since The main traffic I want to isolate to the INT subnet is the database interactions. Could I also explicitly route the INT_002 traffic (has the DB) over that gateway and declare the DMZ_001 as the general default? – Paul Hardwick Jul 22 '14 at 12:04
sure, you can leave main routing table as is. one problem is that all your traffic will go from one of the IPs and will be processed by one of the routing tables above, so if you want to route something extra - edit those tables. – stimur Jul 22 '14 at 14:18
Still doing some experimenting and digging, but your critical comment was about only having a single default gateway. I haven't had a chance to test the route add commands but just taking out that extra gateway got me my basics. I also then consciously realized something about assigning a subnet to one of the ethernet interfaces without the gateway. Those were the only IP-addresses that that interface could access. Made perfect sense but I hadn't consciously realized it. Luckily our entire server collection currently fits in that subnet. Thanks – Paul Hardwick Jul 22 '14 at 22:36
It will help the community if you +1 and accept the answer, so others know it helped :) – stimur Jul 23 '14 at 17:51
Sorry about that. I had tried the up arrow to indicate positive but it had said I didn't have enough reputation yet. But doing check worked. Thanks again – Paul Hardwick Jul 23 '14 at 18:00

Trying to setup Multi-Homing under Ubuntu12 but can only get 1 of 2 IP to accept requests

1 Answers1