4

I have OpenVPN Access Server running on AWS. Here's the configuration: 

172.18.16.0/20
Client (172.18.16.101) ----- OpenVPN Server (172.16.0.0/20)
                                     |
                                     |
                                     |
                                     |
                               Private Subnet (172.16.16.0/20)

The client can connect to the OpenVPN server. The OpenVPN server can ping and access resources on both the client and also on the private subnet. The client can also access all the resources on the OpenVPN server as well as the private subnet. Additionally, the OpenVPN server can also access resources on the client. However, the boxes on the private subnet can't seem to ping or access resources on the client at all.

OpenVPN Server Network Config:

as0t0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:172.18.0.1  P-t-P:172.18.0.1  Mask:255.255.248.0
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:200 
          RX bytes:1223 (1.2 KB)  TX bytes:968 (968.0 B)

as0t1     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:172.18.8.1  P-t-P:172.18.8.1  Mask:255.255.248.0
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:200 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 06:e2:83:cf:4f:27  
          inet addr:172.16.12.204  Bcast:172.16.15.255  Mask:255.255.240.0
          inet6 addr: fe80::4e2:83ff:fecf:4f27/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:1355 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1193 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:124194 (124.1 KB)  TX bytes:153022 (153.0 KB)

OpenVPN Server's Route Table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.0.1      0.0.0.0         UG    0      0        0 eth0
172.16.0.0      0.0.0.0         255.255.240.0   U     0      0        0 eth0
172.18.0.0      0.0.0.0         255.255.248.0   U     0      0        0 as0t0
172.18.8.0      0.0.0.0         255.255.248.0   U     0      0        0 as0t1
172.18.16.101   0.0.0.0         255.255.255.255 UH    0      0        0 as0t0

A Box on the Private Subnet's Route Table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.16.1     0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.16.16.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0

I ran tcpdump on the OpenVPN server box's as0t1 interface while pinging from a box on the private subnet to the client. It looks like the requests was forwarded to the client, and the client also sent back responses (seen on the OpenVPN box). However, it seems that it stops there and the responses never got put back on eth0. I feel that the routing table should be correct as the OpenVPN server can reach everything and the client can also reach the private subnet. I thought the issue was at iptables, but after spending hours looking at it, I am starting to give up. Here's the tables:

Filter Table

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_ACCEPT  all  --  anywhere             anywhere            
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:openvpn
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:https
AS0_WEBACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_WEBACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:943

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_OUT_S2C  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
AS0_OUT_LOCAL  all  --  anywhere             anywhere            

Chain AS0_ACCEPT (5 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_IN (3 references)
target     prot opt source               destination         
ACCEPT     icmp --  anywhere             ip-172-18-0-1.ap-southeast-1.compute.internal  icmp echo-request
ACCEPT     icmp --  anywhere             ip-172-18-8-1.ap-southeast-1.compute.internal  icmp echo-request
AS0_U_TMBDLP_IN  all  --  ip-172-18-16-101.ap-southeast-1.compute.internal  anywhere            
AS0_IN_POST  all  --  anywhere             anywhere            

Chain AS0_IN_NAT (1 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x8000000
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_IN_POST (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             ip-172-16-0-0.ap-southeast-1.compute.internal/16 
AS0_OUT    all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain AS0_IN_PRE (2 references)
target     prot opt source               destination         
AS0_IN     all  --  anywhere             ip-192-168-0-0.ap-southeast-1.compute.internal/16 
AS0_IN     all  --  anywhere             ip-172-16-0-0.ap-southeast-1.compute.internal/12 
AS0_IN     all  --  anywhere             ip-10-0-0-0.ap-southeast-1.compute.internal/8 
DROP       all  --  anywhere             anywhere            

Chain AS0_IN_ROUTE (0 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000000
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_OUT (2 references)
target     prot opt source               destination         
AS0_U_TMBDLP_OUT  all  --  anywhere             ip-172-18-16-101.ap-southeast-1.compute.internal 
AS0_OUT_POST  all  --  anywhere             anywhere            

Chain AS0_OUT_LOCAL (1 references)
target     prot opt source               destination         
DROP       icmp --  anywhere             anywhere             icmp redirect
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_OUT_POST (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain AS0_OUT_S2C (1 references)
target     prot opt source               destination         
AS0_OUT    all  --  anywhere             anywhere            

Chain AS0_U_TMBDLP_IN (1 references)
target     prot opt source               destination         
AS0_IN_NAT  all  --  anywhere             ip-172-16-0-0.ap-southeast-1.compute.internal/16 
AS0_IN_POST  all  --  anywhere             anywhere            

Chain AS0_U_TMBDLP_OUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  ip-172-16-0-0.ap-southeast-1.compute.internal/16  anywhere            
ACCEPT     all  --  ip-172-18-0-0.ap-southeast-1.compute.internal/20  anywhere            
ACCEPT     all  --  ip-172-18-16-0.ap-southeast-1.compute.internal/20  anywhere            
AS0_OUT_POST  all  --  anywhere             anywhere            

Chain AS0_WEBACCEPT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere

NAT Table

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
AS0_NAT_PRE_REL_EST  all  --  anywhere             anywhere             state RELATED,ESTABLISHED

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
AS0_NAT_POST_REL_EST  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_NAT_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000

Chain AS0_NAT (3 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere            

Chain AS0_NAT_POST_REL_EST (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_NAT_PRE (1 references)
target     prot opt source               destination         
AS0_NAT    all  --  anywhere             anywhere             mark match 0x8000000/0x8000000
AS0_NAT_TEST  all  --  anywhere             ip-192-168-0-0.ap-southeast-1.compute.internal/16 
AS0_NAT_TEST  all  --  anywhere             ip-172-16-0-0.ap-southeast-1.compute.internal/12 
AS0_NAT_TEST  all  --  anywhere             ip-10-0-0-0.ap-southeast-1.compute.internal/8 
AS0_NAT    all  --  anywhere             anywhere            

Chain AS0_NAT_PRE_REL_EST (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_NAT_TEST (3 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             mark match 0x4000000/0x4000000
ACCEPT     all  --  anywhere             ip-172-18-0-0.ap-southeast-1.compute.internal/20 
ACCEPT     all  --  anywhere             ip-172-18-16-0.ap-southeast-1.compute.internal/20 
AS0_NAT    all  --  anywhere             anywhere

I am almost sure that the issue is in the iptables rules, but I cannot seem to track it down. Just for some extra fun, I am also throwing the output of iptables trace (captured on the OpenVPN box) here.

Box on Private Subnet Accessing MySQL Server on the Client (failed):

Aug 28 17:42:33 localhost kernel: [21906.075591] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075603] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075608] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075616] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075620] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075625] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:33 localhost kernel: [21906.075629] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2522 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F60F3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115440] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115452] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115458] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115466] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115470] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115475] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.115480] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2523 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9EE3250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175467] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175474] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175479] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175486] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175490] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175494] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:34 localhost kernel: [21907.175498] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2524 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A0265F9F83250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.077995] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078007] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078012] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078021] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078025] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078030] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.078034] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2525 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026601C23250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287922] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287932] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287937] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287945] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287949] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287954] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:36 localhost kernel: [21909.287958] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2526 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026602903250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165296] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165308] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165313] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165321] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165326] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165330] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.165335] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2527 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026611653250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285405] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285414] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285419] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285427] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285431] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285435] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:40 localhost kernel: [21913.285440] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2528 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026612303250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097914] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097926] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097931] TRACE: mangle:AS0_MANGLE_PRE_REL_EST:return:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097939] TRACE: mangle:FORWARD:policy:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097943] TRACE: filter:FORWARD:rule:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097948] TRACE: filter:AS0_ACCEPT:return:1 IN=as0t1 OUT=eth0 MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:48 localhost kernel: [21921.097964] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2529 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026630B93250188401030308) 
Aug 28 17:42:49 localhost kernel: [21921.527746] TRACE: raw:PREROUTING:policy:2 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2530 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026632383250188401030308) 
Aug 28 17:42:49 localhost kernel: [21921.527756] TRACE: mangle:PREROUTING:rule:1 IN=as0t1 OUT= MAC= SRC=172.18.16.101 DST=172.16.22.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=2530 PROTO=TCP SPT=3306 DPT=47378 SEQ=4120644767 ACK=110803956 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020404F40402080A026632383250188401030308) 
##### the rest are cut due to exceeding number of characters allowed #####

Forwarding config

$ cat /proc/sys/net/ipv4/ip_forward
1
keerati
  • 61
  • 7
  • What does `cat /proc/sys/net/ipv4/ip_forward` say? The traces you've got are incomplete; there should be something after `filter:AS0_ACCEPT:return` entries in each case, showing the filter-table disposition in each case. Also, pcaps at the router would be useful. – womble Aug 28 '15 at 23:38
  • cat /proc/sys/net/ipv4/ip_forward returns 1. I also edited the post for more lines from the trace. This is all I can put in due to the 30,000 characters limit. If you need more, I will remove some unneeded characters or will link out to an external file. I will admit that I am not very strong when it comes to packet tracing. I am using AWS, so I am not sure if/how I can pcap at the router. Do you think there's an issue with the router's route table? I did consider that but felt that it should be okay because it seems to know the route to all subnets. Thanks for the help! – keerati Aug 29 '15 at 02:14
  • With EC2 in the mix, you'll need to check the security groups involved to see if they're setup correctly, too. – womble Aug 29 '15 at 05:34
  • The security groups are open (accept all) for testing purpose – keerati Aug 29 '15 at 06:16

1 Answers1

0

TL;DR version:

  1. On the EC2 Dashboard, select the EC2 instance running OpenVPN Access Server
  2. Click Actions -> Networking -> Change Source/Dest. Check -> Yes, Disable

For more information: Disabling Source/Destination Checks

It turns out that all my OpenVPN and route settings were correct, and it was an issue specific to AWS. I stumbled upon the answer while googling specifically for setting up OpenVPN on AWS:

Connecting an EC2 VPC with OpenVPN all routed traffic being lost

What happened to him (quoted below) was exactly the same thing that I am seeing..

However here's the kicker. Doing a tcpdump on the EC2 OpenVPN server shows all traffic flowing as it should:

[root@ip-10-2-0-10 ~]# tcpdump -i eth0 -n host 10.1.0.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:58.779826 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 1, length 64
13:46:58.852756 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 1, length 64
13:46:59.787467 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 2, length 64
13:46:59.847424 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 2, length 64
Community
  • 1
keerati
  • 61
  • 7