Connecting an EC2 VPC with OpenVPN all routed traffic being lost

Question

I'm trying to use OpenVPN on Amazon Linux to connect the local LAN to a VPC on EC2. I have traffic flowing from the OpenVPN instance to any machine on the LAN, but other machines on the VPC are not seeing traffic from the LAN.

Here's a trimmed version of the network:

       Local network          /        EC2 VPC, 10.2.0.*/255.255.255.0
  10.1.0.*/               ,'
     255.255.255.0     ,'
                      .'
                      |
 +---------------+   |             
 | OpenVPN on    |   |             
 |   firewall    XXX |   +---------------+
 | 10.1.0.1      |  XXXX | OpenVPN server|
 +------`.-------+   \  XX  10.2.0.10  `-.    +-------------+
          \          '.  +---------------+  `-. Second server
  +--------`.---+     |                       | 10.2.0.12   |
  |Local server |      \                      +-------------+
  | 10.1.0.3    |       \
  +-------------+        \

From the EC2 OpenVPN server:

[root@ip-10-2-0-10 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.2.0.1        0.0.0.0         UG    0      0        0 eth0
10.1.0.0        10.1.2.5        255.255.255.0   UG    0      0        0 tun0
10.1.2.0        10.1.2.5        255.255.255.0   UG    0      0        0 tun0
10.1.2.5        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

[root@ip-10-2-0-10 ~]# ping -c 2 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.
64 bytes from 10.1.0.3: icmp_seq=1 ttl=127 time=488 ms
64 bytes from 10.1.0.3: icmp_seq=2 ttl=127 time=54.9 ms

--- 10.1.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1056ms
rtt min/avg/max/mdev = 54.925/271.767/488.609/216.842 ms

From the LAN firewall:

root@firewall:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xxx.xxx.xxx.xxx 0.0.0.0         UG    0      0        0 eth0.2
10.1.0.0        0.0.0.0         255.255.255.0   U     0      0        0 br-lan
10.1.2.0        10.1.2.2        255.255.255.0   UG    0      0        0 tun0
10.1.2.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.2.0.0        10.1.2.2        255.255.255.0   UG    0      0        0 tun0

From the Second server on EC2:

[root@ip-10-2-0-12 ~]# ping -c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.

--- 10.1.0.3 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 12014ms

However here's the kicker. Doing a tcpdump on the EC2 OpenVPN server shows all traffic flowing as it should:

[root@ip-10-2-0-10 ~]# tcpdump -i eth0 -n host 10.1.0.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:58.779826 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 1, length 64
13:46:58.852756 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 1, length 64
13:46:59.787467 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 2, length 64
13:46:59.847424 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 2, length 64

Tcpdump on the Second server on EC2 only shows the echo request and no replies.

• IPtables firewalls are disabled on all EC2 instances.
• IP forwarding has been enabled on the OpenVPN server.
• The EC2 security group that both the OpenVPN server and Second server are set to allow all traffic from 10.0.0.0/8 and the network ACLs are wide open.

It appears this traffic is being dropped by Amazon that originates outside the VPC, regardless of the Security Group settings.

Anything I am missing that would be causing this traffic to be dropped?

Answer 1

Turns out EC2 was blocking the forwarded packets from the OpenVPN server.

There is a setting on the EC2 Dashboard under Network & Security -> Network Interfaces -> Actions -> Change Source/Dest. Check.

When I disabled this on all my instances traffic through the VPN is flowing as intended.

Hopefully this will help someone else.

Answer 2

You beat me to your own answer, so I'll expand on it a little...

If you need to build this into your instance bootstrap script, you can add the following to the User Data block (when starting the instance, it's under step 3, Configure Instance Details...expand the Advanced Details section at the bottom of the page) for the instance...update line 3 for correct region:

#!/bin/bash
export my_instance=`curl http://169.254.169.254/latest/meta-data/instance-id`
export AWS_DEFAULT_REGION=us-east-1
aws ec2 modify-instance-attribute --instance-id $my_instance --no-source-dest-check

...and if it's for a NAT instance, you can modify your route table by adding this line (update route-table-id for the route table id used by your VPC):

aws ec2 replace-route --route-table-id rtb-01234567 --destination-cidr-block 0.0.0.0/0 --instance-id $my_instance

For this to work, your instance will need rights to update the attribute and replace the route...You can make that happen with an EC2 Role with the following policy statement, then specify the policy name on the IAM Role box (also in the EC2 wizard's Step 3, Configure Instance Details):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1413415456000",
      "Effect": "Allow",
      "Action": [
        "ec2:ModifyInstanceAttribute",
        "ec2:ReplaceRoute"
      ],
      "Resource": "*"
    }
  ]
}