Secondary IP (eth0:0) acts like main server IP

Question

I have a CentOS server, configured with 4 consecutive IPs:
eth0 5.x.x.251
eth0:0 5.x.x.252
eth0:1 5.x.x.253
eth0:2 5.x.x.254

The problem is that all traffic goes out to the internet with eth0:0 (5.x.x.252) as the source IP, instead of eth0.
# curl ifconfig.me 5.x.x.252

How can I fix this, so that all traffic goes out via eth0, ie my main IP?

PS: My server is VPS running on a Xen dom0, the latter being configured in routed mode networking.

Thanks in advance!

Server configuration

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.251  Bcast:5.x.x.255  Mask:255.255.255.255
          inet6 addr: fe80::x:x:x:x/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14675569 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9463227 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4122016502 (3.8 GiB)  TX bytes:25959110751 (24.1 GiB)
          Interrupt:23 

eth0:0    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE   
          inet addr:5.x.x.252  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:1    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.253  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:2    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.254  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23

# cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
5.x.x.251       [fqdn] [hostname]

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.251
NETMASK=255.255.255.224
SCOPE="peer 5.x.y.82"

# cat ifcfg-eth0:0
DEVICE=eth0:0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.252
NETMASK=255.255.255.224

# cat route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
5.x.y.82        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
5.x.x.224       0.0.0.0         255.255.255.224 U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         5.x.y.82        0.0.0.0         UG        0 0          0 eth0

Update 29 August 2012
When I run /etc/init.d/network restart I get an RTNETLINK answers: File exists error, which leads me to believe that there's another route to another network using the default gateway, as seen here.

# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]

The sure thing is that when I first delete ip route del default via 5.x.y.82 dev eth0 and then add it again with the .251 IP, everything works as expected.
Is there any way to add a route del command either to the ifcfg-eth0 or the route-eth0 files? I can see that this can be done with pre-up ip addr del ... on Debian (more info), but what about CentOS?

Here's the output of `ip ro li`: http://pastebin.com/A1KRiWTz I think the last line is what I'm looking for... how can I change `src 5.x.x.252` to `scr 5.x.x.251`? — George Tasioulis, Aug 28 '12 at 22:59
Are you using an official CentOS kernel? Try "uname -a". I've seen similarly weird behavior (e.g., arp flux) with OpenVZ kernels. — joechip, Sep 07 '12 at 05:35
I'm giving the correct answer & bounty to Zoredache who was the one to find the source of the problem in the first place. With his help I managed to at least fix it on the live system, and upon restart (via `/etc/rc.local`). Only thing left to fix is when I run `/etc/init.d/network restart` but ok I can live without it :) — George Tasioulis, Sep 07 '12 at 09:19

score 6 · Accepted Answer · answered Aug 28 '12 at 23:13

6

From this document it appears that at least on CentOS 5 and I suspect above you can specify your routes by simply giving the correct iproute2 command arguments. (see the section on the IP Command Arguments Format)

So instead of writing something like:

# route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

You could just have a file like this:

default via 5.x.y.82 dev eth0  src 5.x.x.251

I don't have a Redhat/Redhat derived box handy to test though.

answered Aug 28 '12 at 23:13

Zoredache

128,755
40
271
413

I just tried it, unfortunately even when defining the .251 IP and restarting networking, `ip ro li` still reports the .252 as source. – George Tasioulis Aug 28 '12 at 23:19
1

How annoying. If you remove the default route and add it back manually do things work properly? `ip route del default via 5.x.y.82 dev eth0; ip route add default via 5.x.y.82 dev eth0 src 5.x.x.251`? – Zoredache Aug 28 '12 at 23:22
Let me login via console from the dom0 to try this out and i'll get back to you in a second. – George Tasioulis Aug 28 '12 at 23:23
Yay! deleting the ip default route and adding it back works, BUT if I restart the VM or do a `/etc/init.d/network restart` I'm back to .252 :( I guess could add the above command to `/etc/rc.local` to cover the VM restart part... – George Tasioulis Aug 28 '12 at 23:30
Hrm, well it sure seems from the documentation that the route-interface file should support any valid iproute command. You may need to either go with the rc.local fix, or hope someone else can give you an answer. – Zoredache Aug 29 '12 at 00:37
When I tried adding the `ip route del ...` command to `route-eth0` and restarted network, I got the following error `Error: an inet prefix is expected rather than "ip".` Any other idea how I can delete the default route either via the `ifcfg-eth0` or the `route-eth0` file? (PS: I have updated my question with some additional info) – George Tasioulis Aug 29 '12 at 21:05
+1 for the answer. Please read official docs, route-eth0 shouldn't contain these fields. – GioMac Sep 04 '12 at 14:50

Panagiotis Moustafellos · Answer 2 · 2012-08-29T21:31:03.123

2

First clear your 0.0.0.0 default gateway

route del -net default

Then declare default eth0 as default gateway device

route add -net default gw 5.x.y.82 dev eth0

It should work. Verify with

ip ro li

To save that as a rule try this messy thing:

Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" .

What happens is that the network-functions script decides which device gets to be the default gateway. It examines the route to the gateway IP with ip get route to GATEWAY and, using sed, matches the a device. It seems that eth0:0 gets to be the one each time, so by hardcoding GATEWAYDEV="eth0" you make sure that eth0 is always selected as the gateway device.

In older versions of RH you could just edit /etc/sysconfig/static-routes and add default via 5.x.y.82 dev eth0

As a last resort /etc/rc.local is always your friend!

edited Aug 29 '12 at 21:31

answered Aug 28 '12 at 23:50

Panagiotis Moustafellos

2,408
1
12
17

Kalispera :) Zoredache already proposed this as a solution and it partially works. What I mean is that it will work fine until I either reboot the VM or run `/etc/init.d/network restart`. Then I'm back to the .252 IP – George Tasioulis Aug 28 '12 at 23:57
On older RH versions you could add at `/etc/syconfig/static-routes` a `default gw 5.x.y.82 dev eth0` On newer versions edit `/etc/sysconfig/network-scripts/ifcfg-eth0` and `/etc/sysconfig/network-scripts/ifcfg-eth0:0`, remove from eth0:0 the `DEFROUTE=yes` and add it to eth0, I am also editing this on my reply above. – Panagiotis Moustafellos Aug 29 '12 at 00:24
I tried the `DEFROUTE=yes` thing on `ifcfg-eth0` (also tried adding `DEFROUTE=no` to the other `ifcfg-eth0:?` files, but it didn't work. PS: I have updated my question with some extra information – George Tasioulis Aug 29 '12 at 21:07
What I am about to suggest is a bit messy but does the trick. Open `/etc/sysconfig/network-scripts/network-functions`, locate function `add_default_route ()` and in that function, after the call to `find_gateway_dev` add a line with `GATEWAYDEV="eth0"` Placing that in my answer. – Panagiotis Moustafellos Aug 29 '12 at 21:15
Unfortunatelly this didn't do the trick either... :( Here's a [screenshot](http://cl.ly/J6YN). I got the `RTNETLINK answers: File exists` error again. Until now the only thing that worked is manually deleting the default route and adding it again. This `File exists` error occurs because of two default routes. Does my `route -n` help? – George Tasioulis Aug 29 '12 at 21:33
I am assuming that for this "hack" to take place, first you 'll have to `route del -net default` and then `/etc/init.d/network reload`. Reload once more to see if the error pops up. If not, you are good to go. – Panagiotis Moustafellos Aug 29 '12 at 21:36
Tried that too. I connected to the VPS via console, did the `ip route del default ...` first, then restarted `/etc/init.d/network` two times to check. Same error both times (`RTNETLINK answers: File exists`). – George Tasioulis Aug 29 '12 at 21:50
Hmm, can't think of something right now, but this has to do with `network-functions`, check it out in extend. Till you or someone else figures it out, append at the end of that script the `ip route del default` and `ip route add default gw x.x.x.x dev eth0` with an `echo "WARNING: $(basename "$0") eth0 is hardcoded as default gw interface"` to save some headaches in the future – Panagiotis Moustafellos Aug 29 '12 at 21:58

score 2 · Answer 3 · answered Aug 28 '12 at 23:59

2

If the aliased IP addresses are not to be used as source addresses to non-local destinations, they should not be in the same subnet as the target of your default route. So change their netmasks to 255.255.255.255 and remove their broadcast addresses.

answered Aug 28 '12 at 23:59

David Schwartz

31,215
2
53
82

score 2 · Answer 4 · answered Sep 05 '12 at 17:21

Sounds like you might be a victim of a setting in newer kernels. Enabling alias promotion will get past that.

Set this in your /etc/sysctl.conf

net.ipv4.conf.all.promote_secondaries = 1

Incidentally, do you have any funky SNAT/MASQUERADE iptables rules that are altering the outbound source IP?

score 1 · Answer 5 · edited Apr 13 '17 at 12:14

1

Since they are all on the same subnet the kernel will pick one to use.. You can verify this by running

 ip addr list dev eth0

You have to do some iptables magic to get it to work correctly. See the following

When IP aliasing how does the OS determine which IP address will be used as source for outbound TCP/IP connections?

edited Apr 13 '17 at 12:14

Community

1

answered Aug 28 '12 at 22:32

Mike

21,910
7
55
79

Here's the output of `ip addr list dev eth0`: http://pastebin.com/mFv4XCb9 I understand that it can be fixed with iptables, but my main concern first of all is WHY this is happening? I just noticed that the eth0 subnet mask from `ifconfig` compared to `ifcfg-eth0` is different... – George Tasioulis Aug 28 '12 at 22:56

score 0 · Answer 6 · answered Sep 05 '12 at 14:39

route and ifconfig use old, deprecated, network interfaces in kernel that don't know about policy based routing. You should be using the ip command, especially for routing-related information.

That being said, ip ro li or the full command ip route list table main, outputs the actual routing table used by the kernel. ip rule list will show you all the routing tables in use (there should at least be local, main and default).

As you've posted (from http://pastebin.com/A1KRiWTz):

# ip ro li
5.x.y.82 dev eth0  proto kernel  scope link  src 5.x.x.251 
5.x.x.224/27 dev eth0  proto kernel  scope link  src 5.x.x.252 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 5.x.y.82 dev eth0  src 5.x.x.252

you need to remove the default route and recreate it with correct source address:

ip route del default 
ip route add default via 2.x.y.82 dev eth0 src 5.x.x.251 scope global

I don't know how to do it the RedHat way though (where to put it in RedHat-specific config files), using /etc/rc.local should work, albeit it's somewhat hacky solution. You could try using system-config-network

score -3 · Answer 7 · answered Sep 05 '12 at 14:01

-3

You need to write a PREROUTING rule for inbound and outbound traffic in iptable, you need to say any tcp or UDP traffic or port 80 or 8080 (for default internet only) should go to desired eth.

answered Sep 05 '12 at 14:01

RL89

101

Secondary IP (eth0:0) acts like main server IP

7 Answers7

Linked