Linux IP Alias not working - Amazon EC2

Question

I can't seem to figure out this IP aliasing on Amazon EC2. I know it should be straight fwd though.

In summary I have two questions (my scenario details follows after questions), in priority:

How to get the routing working via the CLI commands?
Then after [1] works, how to make the configs stick via config files so that it sticks even after reboots?

The config files is my secondary problem, seeing that I can't even get the routing going via CLI.

Here is what I have by default:

eth0      Link encap:Ethernet  HWaddr 0a:64:bd:67:d6:4a  
          inet addr:172.31.16.15  Bcast:172.31.31.255  Mask:255.255.240.0
          inet6 addr: fe80::864:bdff:fe67:d64a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:266 errors:0 dropped:0 overruns:0 frame:0
          TX packets:257 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:29714 (29.7 KB)  TX bytes:29843 (29.8 KB)

With the following routing table:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.16.1     0.0.0.0         UG    0      0        0 eth0
172.31.16.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0

What I want is:

eth0 -> 172.31.16.15
eth0:0 -> 172.31.16.100
eth0:1 -> 172.31.16.101

With the correct routing of course (and I think this is where things go wrong for me) so that I can successfully accomplish this:

1.  telnet -b 172.31.16.15 172.31.16.20 5222
2.  telnet -b 172.31.16.100 172.31.16.20 5222
3.  telnet -b 172.31.16.101 172.31.16.20 5222

Even pinging works only from the 172.31.16.15 ip:

1.  ping -I 172.31.16.15 172.31.16.20
2.  ping -I 172.31.16.100 172.31.16.20
3.  ping -I 172.31.16.101 172.31.16.20

Only [1] works for both the telnet and ping commands above.

When I do the telnet command, and I tcpdump the traffic, the results as follows:

For 172.31.16.15 when it works:

12:58:14.082176 IP (tos 0x10, ttl 64, id 59547, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.15.26798 > 172.31.16.20.5222: Flags [S], cksum 0x7890 (incorrect -> 0x455e), seq 2790518412, win 29200, options [mss 1460,sackOK,TS val 2360855 ecr 0,nop,wscale 7], length 0
12:58:14.082848 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.20.5222 > 172.31.16.15.26798: Flags [S.], cksum 0xfb9b (correct), seq 1051320718, ack 2790518413, win 28960, options [mss 1460,sackOK,TS val 2304582 ecr 2360855,nop,wscale 7], length 0
12:58:14.082877 IP (tos 0x10, ttl 64, id 59548, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.16.15.26798 > 172.31.16.20.5222: Flags [.], cksum 0x7888 (incorrect -> 0x9aa3), ack 1, win 229, options [nop,nop,TS val 2360855 ecr 2304582], length 0

For 172.31.16.100 when it doesn't work (also, nothing arrives at receiving end):

12:59:01.001723 IP (tos 0x10, ttl 64, id 45034, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xf906), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2372585 ecr 0,nop,wscale 7], length 0
12:59:02.000831 IP (tos 0x10, ttl 64, id 45035, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xf80c), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2372835 ecr 0,nop,wscale 7], length 0
12:59:04.004827 IP (tos 0x10, ttl 64, id 45036, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xf617), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2373336 ecr 0,nop,wscale 7], length 0
12:59:08.012822 IP (tos 0x10, ttl 64, id 45037, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xf22d), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2374338 ecr 0,nop,wscale 7], length 0
12:59:16.036831 IP (tos 0x10, ttl 64, id 45038, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xea57), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2376344 ecr 0,nop,wscale 7], length 0
12:59:32.068840 IP (tos 0x10, ttl 64, id 45039, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.16.100.17006 > 172.31.16.20.5222: Flags [S], cksum 0x78e5 (incorrect -> 0xdaaf), seq 1028496387, win 29200, options [mss 1460,sackOK,TS val 2380352 ecr 0,nop,wscale 7], length 0

I have tried this in /etc/network/interfaces:

auto eth0:0
iface eth0:0 inet static
address 172.31.16.100
netmask 255.255.240.0
broadcast 172.31.31.255
network 172.31.16.0
gateway 172.31.16.1

auto eth0:1
iface eth0:1 inet static
address 172.31.16.101
netmask 255.255.240.0
broadcast 172.31.31.255
network 172.31.16.0
gateway 172.31.16.1

When I restart networking it doesn't take effect. Also when I reboot machine I can't ssh into it again either. Seems something takes effect then, but obviously very wrong.

I've also done the CLI sudo ifconfig way:

$ sudo ifconfig eth0:0 172.31.16.100 netmask 255.255.240.0 broadcast 172.31.31.255 up
$ sudo ifconfig eth0:1 172.31.16.101 netmask 255.255.240.0 broadcast 172.31.31.255 up

where my IP alias do take effect immediately:

$ ifconfig 
eth0      Link encap:Ethernet  HWaddr 0a:64:bd:67:d6:4a  
          inet addr:172.31.16.15  Bcast:172.31.31.255  Mask:255.255.240.0
          inet6 addr: fe80::864:bdff:fe67:d64a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1224 errors:0 dropped:0 overruns:0 frame:0
          TX packets:943 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:93498 (93.4 KB)  TX bytes:118463 (118.4 KB)

eth0:0    Link encap:Ethernet  HWaddr 0a:64:bd:67:d6:4a  
          inet addr:172.31.16.100  Bcast:172.31.31.255  Mask:255.255.240.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:1    Link encap:Ethernet  HWaddr 0a:64:bd:67:d6:4a  
          inet addr:172.31.16.101  Bcast:172.31.31.255  Mask:255.255.240.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

with the routing table still looking the same:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.16.1     0.0.0.0         UG    0      0        0 eth0
172.31.16.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0

but again, I can't do the telnet as described above for [2] and [3].

Also, after entering following commands (and flushing the routing tables):

echo 200 EJ0 >> /etc/iproute2/rt_tables
echo 201 EJ1 >> /etc/iproute2/rt_tables
ip route add 172.31.16.0 dev eth0:0 src 172.31.16.100 table EJ0
ip route add default via 172.31.16.1 table EJ0
ip route add 172.31.16.0 dev eth0:1 src 172.31.16.101 table EJ1
ip route add default via 172.31.16.1 table EJ1
ip route add 172.31.16.0 dev eth0:0 src 172.31.16.100
ip route add 172.31.16.0 dev eth0:1 src 172.31.16.101
ip rule add from 172.31.16.100 table EJ0
ip rule add from 172.31.16.101 table EJ1

the ping and telnet commands still don't work.

More info:

$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0a:64:bd:67:d6:4a brd ff:ff:ff:ff:ff:ff
    inet 172.31.16.15/20 brd 172.31.31.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.31.16.100/20 brd 172.31.31.255 scope global secondary eth0:0
       valid_lft forever preferred_lft forever
    inet 172.31.16.101/20 brd 172.31.31.255 scope global secondary eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::864:bdff:fe67:d64a/64 scope link 
       valid_lft forever preferred_lft forever

and

$ ip route show
default via 172.31.16.1 dev eth0 
172.31.16.0 dev eth0  scope link  src 172.31.16.100 
172.31.16.0/20 dev eth0  proto kernel  scope link  src 172.31.16.15

This is all so that HAProxy can successfully connect to an ejabberd instance but doing it from two different src IPs (eth0:0 and eth0:1).

Any advice most welcome and tons appreciated.

IIRC ip aliases only have the address and broadcast values set and omit the broadcast, network and gateway in `/etc/network/interfaces`. — HBruijn, Jun 12 '14 at 10:52
For now my 2nd worry will be to get the configuration working with the actual config files. I can get things going via CLI, getting the aliases there, but my routing is not working. Will edit original post/question accordingly. — 0v3rst33r, Jun 12 '14 at 11:00
You have 192.168.1.x addresses in the `/etc/network/interfaces` you show, but you manually set up 172.31.16.x ?? — Benoit, Jun 12 '14 at 11:08
Don't you have **iptables** rules configured ? (use `iptables -L` to check) — Benoit, Jun 12 '14 at 11:14
After you perform your ping and / or telnet, it would be interesting to run `arp -an` and publish the output. — Benoit, Jun 12 '14 at 11:21
Sorry the 192.168.1.x addresses was just a typo, thank you for pointing it out and I fixed it in my original post. — 0v3rst33r, Jun 12 '14 at 11:31
The problem might be on the other end of the connection. It might be that the packets are received by the other end, but it is unable to deliver replies back. What packets are actually send on the network? — kasperd, Jun 12 '14 at 11:51
@kasperd I included the tcpdump in original post. Looks like for the other IPs the machine is definitely trying to push out the initial telnet SYN packet but it is never ACKd from the other end. Will investigate further, might have been just a wild goose chase and packets not even allowed due to firewall etc. Thanks for all feedback thus far. — 0v3rst33r, Jun 12 '14 at 13:07
It isn't that, when I spin up another EC2 instance with the 172.31.16.100 address on its eth0 NIC it can connect to 172.31.16.20 just fine. — 0v3rst33r, Jun 12 '14 at 13:13
You just write "EC2 instance". Does it means that all these servers you're talking about are running inside the Amazon cloud and are therefore virtual machines ? — Benoit, Jun 12 '14 at 14:17
@c0rn0 It is not just trying to send the SYN packet. It is actually succeeding in sending the SYN packet. Look at `netstat -nt` output on the receiving end to see if the connections are there in `SYN_RECV` state? It might be the SYN-ACK packets are being routed in the wrong direction. — kasperd, Jun 12 '14 at 15:50
@kasperd sorry, I spoke wrong: yes, it sent the SYN flag, which TCP is suppose to do :) However, I did a tcpdump on the receiving end as well, nothing arrived there though. — 0v3rst33r, Jun 12 '14 at 15:59
@c0rn0 Are the SYN packets being sent to the same MAC address in both cases? Does it behave any differently if you forget about the routing policies? I am not convinced routing policies are the correct solution for your use case. — kasperd, Jun 12 '14 at 16:27
I have come to realize now that more than likely one is not even able to do IP aliasing at Amazon EC2. Perhaps this virtual IP that I am trying to setup and use is just a futile exercise. Will explore more in the meantime. — 0v3rst33r, Jun 12 '14 at 16:53
Aha! This is not possible with Amazon's EC2 VPC. Sorry everyone for wasting your time. I've update the question's Title. Thank you so much for all advice and asking questions to help me, much appreciated! — 0v3rst33r, Jun 12 '14 at 17:22
I will check out this link though: [Leveraging Multiple IP Addresses for Virtual IP Address Fail-over in 6 Simple Steps](http://aws.amazon.com/articles/2127188135977316). It might be my only option, but the standard Linux IP aliasing on physical NICs is definitely not the way when working on Amazon EC2 VPC. If I have success from said link I will update my question accordingly and/or post the answer. — 0v3rst33r, Jun 12 '14 at 17:28
To correct myself: **IP aliasing is possible with Amazon's EC2 VPC**. Details can be found at [Multiple Private IP Addresses](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html). I've also given this as an official answer to this question. — 0v3rst33r, Jun 12 '14 at 23:11

score 2 · Accepted Answer · answered Jun 12 '14 at 23:00

IP aliasing is indeed possible with Amazon EC2 VPC!!

If you are using Ubuntu Linux (like I am at the moment) you still need to add your IP alias as per usual for Linux BUT the crucial part is also to do the ADDITIONAL configuring of it on the Amazon EC2 console itself as shown here Multiple Private IP Addresses.

Thanks all for your comments and contributions.

score 1 · Answer 2 · answered Jun 12 '14 at 11:04

Similar to routing, when linux finds multiple entities in the same network, it will use the first matching route/interface to get there. In this case, it's eth0 and 172.31.16.15.

To make linux use those aliases as both source addresses, and as completely functional interfaces, you'll need to create multiple routing tables, one for each virtual interface.

echo 200 EJ0 >> /etc/iproute2/rt_tables

echo 201 EJ1 >> /etc/iproute2/rt_tables

Add routes

ip route add 172.31.16.0 dev eth0:0 src 172.31.16.100 table EJ0

ip route add default via 172.31.16.1 table EJ0

ip route add 172.31.16.0 dev eth0:1 src 172.31.16.101 table EJ1

ip route add default via 172.31.16.1 table EJ1

Then tell the main table.

ip route add 172.31.16.0 dev eth0:0 src 172.31.16.100

ip route add 172.31.16.0 dev eth0:1 src 172.31.16.101

then add the rules

ip rule add from 172.31.16.100 table EJ0

ip rule add from 172.31.16.101 table EJ1

Lots of this taken from the ever useful linux policy routing page

I've tried the flushing now, still no luck. I've added more info to my question RE the instructions you gave me. Thanks for all help so far! — 0v3rst33r, Jun 12 '14 at 11:27

Linux IP Alias not working - Amazon EC2

2 Answers2