Routing issue with multiple IP addresses on EC2 (VPC) host

Question

We run a EC2 Ubuntu server acting as a STUN server for the NAT Traversal mechanism in my company's application. By the nature of the STUN protocol the server needs two IP public addresses. We hired some consultants to set this up for us but they seem to have failed because the STUN server is not responding to any external binding requests.

Here's what we know:

We have two EIPs mapped to the server (the same network interface).
The server is running inside a VPC subnet.
If we send the STUN binding requests locally at the STUN server (by specifying one of the EIPs to the STUN client) it succeeds.
If we do the same request from one of our other EC2 servers (outside of the VPC subnet) the request fails.
If we do the same request from desktop computer at our office it fails.
The STUN server is receiving the binding requests but it looks like it fails to route the response. Here's screenshot of Wireshark's parse of a dump file created during a binding request.

I am guessing that we are missing something in our routing tables. Unfortunately IP routing is not one of our core competencies (especially on Linux). Below is the output from ifconfig that hopefully gives you some more useful information.

eth0      Link encap:Ethernet  HWaddr 0e:5a:ec:5d:6d:d5
          inet addr:10.0.0.22  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::c5a:ecff:fe5d:6dd5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28031 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18370 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2329243 (2.3 MB)  TX bytes:19316360 (19.3 MB)
          Interrupt:28

eth0:1    Link encap:Ethernet  HWaddr 0e:5a:ec:5d:6d:d5
          inet addr:10.0.0.23  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:28

eth0:2    Link encap:Ethernet  HWaddr 0e:5a:ec:5d:6d:d5
          inet addr:107.23.130.153  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:28

eth0:3    Link encap:Ethernet  HWaddr 0e:5a:ec:5d:6d:d5
          inet addr:107.23.35.233  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:28

Here's the output from running sudo route:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

default         10.0.0.1        0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth0

NickW · Answer 1 · 2013-03-11T12:14:02.310

10.0.0.1 is your default gateway, is there a router on the 107.23.X.X network?

I have a feeling you're going to need to set up proper routing tables, to ensure packets that come in on one network also go back out through the same interface, on the same network.

You could do something like this:

echo 200 Ten >> /etc/iproute2/rt_tables
ip route add 107.23.130.153 dev eth0:2 src 107.23.130.153 table Ten
ip route add default via whatever table Ten

echo 200 Twenty >> /etc/iproute2/rt_tables
ip route add 107.23.35.233 dev eth0:3 src 107.23.35.233 table Twenty
ip route add default via whatever table Twenty

replace whatever with the gateway for those two addresses, I assume they're Point to Point addresses.

Then add the rules which send traffic from your IP through the correct table, and interfaces.

ip rule add from 107.23.130.53 table Ten
ip rule add from 107.23.35.233 table Twenty

Anyhow, there is no shortage of information available online, look up policy routing

Routing issue with multiple IP addresses on EC2 (VPC) host

1 Answers1