How do i get Strongswan / IPTables to route data back to my road warrior client correctly?

Question

I have a simple VPN. I have a client on 10.185.28.241 who gets a virtual IP of 10.42.42.0/24 from the VPN which is located at 10.112.18.105 and is providing access to machines in the 10.112.0.0/16 CIDR. I'm running StrongSwan U5.3.2/K3.10.0-229.el7.x86_64 on Centos 7. My network layout looks something like this:

[Windows 2012 Client      ]--------[VPN         ]---------[Hidden network]
10.185.28.241 / 10.42.42.1         10.112.18.105           10.112.0.0/16

Yes, everything about this network is internal. The Windows Client is actually in AWS while the 10.112 network is in my colo, but all over a pre-existing site to site VPN. I'm setting this up because I eventually want to route external traffic back through the VPN so it will appear to come through our colo where we're whitelisted by vendors. But I'm not there yet.

I can connect to the VPN from Windows and it's connecting over the very fancy IKEv2. When I ping an IP on the hidden network, I see the packets coming over from tcpdump but the 10.112.18.105 machine refuses to return the response.

So on the Windows machine I'm executing:

C:\temp> ping 10.112.8.102

Pinging 10.112.8.102 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 10.112.8.102:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\temp> ping 10.112.8.102

And on the VPN server I see:

[root@localhost strongswan]# tcpdump -n -i any host 10.112.8.102 or host 10.42.42.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:38:42.616871 IP 10.112.18.105 > 10.112.8.102: ICMP echo request, id 1, seq 161, length 40
17:38:42.617324 IP 10.112.8.102 > 10.112.18.105: ICMP echo reply, id 1, seq 161, length 40
17:38:47.280369 IP 10.42.42.1 > 10.112.8.102: ICMP echo request, id 1, seq 162, length 40
17:38:47.280457 IP 10.112.18.105 > 10.112.8.102: ICMP echo request, id 1, seq 162, length 40
17:38:47.280872 IP 10.112.8.102 > 10.112.18.105: ICMP echo reply, id 1, seq 162, length 40
17:38:52.280116 IP 10.42.42.1 > 10.112.8.102: ICMP echo request, id 1, seq 163, length 40
17:38:52.280213 IP 10.112.18.105 > 10.112.8.102: ICMP echo request, id 1, seq 163, length 40
17:38:52.280672 IP 10.112.8.102 > 10.112.18.105: ICMP echo reply, id 1, seq 163, length 40
17:38:57.279917 IP 10.42.42.1 > 10.112.8.102: ICMP echo request, id 1, seq 164, length 40
17:38:57.280018 IP 10.112.18.105 > 10.112.8.102: ICMP echo request, id 1, seq 164, length 40
17:38:57.280488 IP 10.112.8.102 > 10.112.18.105: ICMP echo reply, id 1, seq 164, length 40
^C
11 packets captured
12 packets received by filter
0 packets dropped by kernel
[root@localhost strongswan]#

The reply is coming back to the VPN server ("10.112.8.102 > 10.112.18.105: ICMP echo reply") but the VPN server isn't passing the response back to the client. I've spent a good 10 hours trying to track this down and all I've found are lots of people with a similar issue :-(.

Some background info:

ipsec.conf:

config setup
    charondebug="ike 2, knl 2, cfg 2, net 2, esp 2, dmn 2,  mgr 2"

conn %default
    keyexchange=ikev2
    ike=aes128-sha256-ecp256,aes256-sha384-ecp384,aes128-sha256-modp2048,aes128-sha1-modp2048,aes256-sha384-modp4096,aes256-sha256-modp4096,aes256-sha1-modp4096,aes128-sha256-modp1536,aes128-sha1-modp1536,aes256-sha384-modp2048,aes256-sha256-modp2048,aes256-sha1-modp2048,aes128-sha256-modp1024,aes128-sha1-modp1024,aes256-sha384-modp1536,aes256-sha256-modp1536,aes256-sha1-modp1536,aes256-sha384-modp1024,aes256-sha256-modp1024,aes256-sha1-modp1024!
    esp=aes128gcm16-ecp256,aes256gcm16-ecp384,aes128-sha256-ecp256,aes256-sha384-ecp384,aes128-sha256-modp2048,aes128-sha1-modp2048,aes256-sha384-modp4096,aes256-sha256-modp4096,aes256-sha1-modp4096,aes128-sha256-modp1536,aes128-sha1-modp1536,aes256-sha384-modp2048,aes256-sha256-modp2048,aes256-sha1-modp2048,aes128-sha256-modp1024,aes128-sha1-modp1024,aes256-sha384-modp1536,aes256-sha256-modp1536,aes256-sha1-modp1536,aes256-sha384-modp1024,aes256-sha256-modp1024,aes256-sha1-modp1024,aes128gcm16,aes256gcm16,aes128-sha256,aes128-sha1,aes256-sha384,aes256-sha256,aes256-sha1!
    dpdaction=clear
    dpddelay=300s
    rekey=no
    left=10.112.18.105
    leftsubnet=10.112.0.0/16
    leftfirewall=yes
    leftcert=vpnHostCert.der
    right=%any
    rightdns=10.112.1.140,10.112.1.141
    rightsourceip=10.42.42.0/24

conn IPSec-IKEv2
    keyexchange=ikev2
    auto=add

conn IPSec-IKEv2-EAP
    also="IPSec-IKEv2"
    rightauth=eap-mschapv2
    rightsendcert=never
    eap_identity=%any

conn CiscoIPSec
    keyexchange=ikev1
    forceencaps=yes
    authby=xauthrsasig
    xauth=server
    auto=add

Strongswan statusall:

[root@localhost strongswan]# strongswan statusall
Status of IKE charon daemon (strongSwan 5.3.2, Linux 3.10.0-229.el7.x86_64, x86_64):
  uptime: 26 minutes, since Apr 15 17:31:18 2016
  malloc: sbrk 1630208, mmap 0, used 515488, free 1114720
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 1
  loaded plugins: charon aes des rc2 sha1 sha2 md4 md5 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp xcbc cmac hmac ctr ccm curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-md5 eap-gtc eap-mschapv2 eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp
Virtual IP pools (size/online/offline):
  10.42.42.0/24: 254/1/0
Listening IP addresses:
  10.112.18.105
  192.168.122.1
Connections:
 IPSec-IKEv2:  10.112.18.105...%any  IKEv2, dpddelay=300s
 IPSec-IKEv2:   local:  [10.112.18.105] uses public key authentication
 IPSec-IKEv2:    cert:  "C=US, O=Aurea, CN=cis-migration-vpn.aes.aurea.com"
 IPSec-IKEv2:   remote: uses public key authentication
 IPSec-IKEv2:   child:  10.112.0.0/16 === dynamic TUNNEL, dpdaction=clear
IPSec-IKEv2-EAP:  10.112.18.105...%any  IKEv2, dpddelay=300s
IPSec-IKEv2-EAP:   local:  [10.112.18.105] uses public key authentication
IPSec-IKEv2-EAP:    cert:  "C=US, O=Aurea, CN=cis-migration-vpn.aes.aurea.com"
IPSec-IKEv2-EAP:   remote: uses EAP_MSCHAPV2 authentication with EAP identity '%any'
IPSec-IKEv2-EAP:   child:  10.112.0.0/16 === dynamic TUNNEL, dpdaction=clear
  CiscoIPSec:  10.112.18.105...%any  IKEv1, dpddelay=300s
  CiscoIPSec:   local:  [10.112.18.105] uses public key authentication
  CiscoIPSec:    cert:  "C=US, O=Aurea, CN=cis-migration-vpn.aes.aurea.com"
  CiscoIPSec:   remote: uses public key authentication
  CiscoIPSec:   remote: uses XAuth authentication: any
  CiscoIPSec:   child:  10.112.0.0/16 === dynamic TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
IPSec-IKEv2-EAP[1]: ESTABLISHED 26 minutes ago, 10.112.18.105[10.112.18.105]...10.185.28.241[10.185.28.241]
IPSec-IKEv2-EAP[1]: Remote EAP identity: ryan
IPSec-IKEv2-EAP[1]: IKEv2 SPIs: 9b0fd7b8f81a8c6c_i 966e28db92550c47_r*, rekeying disabled
IPSec-IKEv2-EAP[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/MODP_1024
[root@localhost strongswan]#

Network Information

[root@localhost strongswan]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).


# VPN
net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
[root@localhost strongswan]# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.112.18.105  netmask 255.255.254.0  broadcast 10.112.19.255
        inet6 fe80::250:56ff:fe9a:b6c  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9a:0b:6c  txqueuelen 1000  (Ethernet)
        RX packets 132983  bytes 33121546 (31.5 MiB)
        RX errors 0  dropped 678  overruns 0  frame 0
        TX packets 62082  bytes 11920649 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 20  bytes 2100 (2.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20  bytes 2100 (2.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:a1:a6:70  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@localhost strongswan]# ip -4 a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 10.112.18.105/23 brd 10.112.19.255 scope global ens192
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
[root@localhost strongswan]# ip -4 r s t 0
default via 10.112.18.1 dev ens192  proto static  metric 100
10.112.18.0/23 dev ens192  proto kernel  scope link  src 10.112.18.105  metric 100
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1
broadcast 10.112.18.0 dev ens192  table local  proto kernel  scope link  src 10.112.18.105
local 10.112.18.105 dev ens192  table local  proto kernel  scope host  src 10.112.18.105
broadcast 10.112.19.255 dev ens192  table local  proto kernel  scope link  src 10.112.18.105
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
broadcast 192.168.122.0 dev virbr0  table local  proto kernel  scope link  src 192.168.122.1
local 192.168.122.1 dev virbr0  table local  proto kernel  scope host  src 192.168.122.1
broadcast 192.168.122.255 dev virbr0  table local  proto kernel  scope link  src 192.168.122.1
[root@localhost strongswan]# ip route show table 220
10.42.42.1 via 10.112.18.1 dev ens192  proto static  src 10.112.18.105
[root@localhost strongswan]# ip xfrm policy show
src 10.42.42.1/32 dst 10.112.0.0/16
        dir fwd priority 2883 ptype main
        tmpl src 10.185.28.241 dst 10.112.18.105
                proto esp reqid 1 mode tunnel
src 10.42.42.1/32 dst 10.112.0.0/16
        dir in priority 2883 ptype main
        tmpl src 10.185.28.241 dst 10.112.18.105
                proto esp reqid 1 mode tunnel
src 10.112.0.0/16 dst 10.42.42.1/32
        dir out priority 2883 ptype main
        tmpl src 10.112.18.105 dst 10.185.28.241
                proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
        socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
        socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
        socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
        socket out priority 0 ptype main
src ::/0 dst ::/0
        socket in priority 0 ptype main
src ::/0 dst ::/0
        socket out priority 0 ptype main
src ::/0 dst ::/0
        socket in priority 0 ptype main
src ::/0 dst ::/0
        socket out priority 0 ptype main
[root@localhost strongswan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.112.18.1     0.0.0.0         UG    100    0        0 ens192
10.112.18.0     0.0.0.0         255.255.254.0   U     100    0        0 ens192
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
[root@localhost strongswan]#

IPTables

[root@localhost strongswan]# iptables-save
# Generated by iptables-save v1.4.21 on Fri Apr 15 18:03:58 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [152:18472]
-A INPUT -p udp -m udp --dport 1701 -j ACCEPT
-A INPUT -p udp -m udp --dport 4500 -j ACCEPT
-A INPUT -p udp -m udp --dport 500 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p esp -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Apr 15 18:03:58 2016
# Generated by iptables-save v1.4.21 on Fri Apr 15 18:03:58 2016
*nat
:PREROUTING ACCEPT [866:126303]
:INPUT ACCEPT [9:2520]
:OUTPUT ACCEPT [14:1198]
:POSTROUTING ACCEPT [14:1198]
-A POSTROUTING -s 10.42.42.0/24 -o ens192 -m policy --dir out --pol ipsec -j ACCEPT
-A POSTROUTING -s 10.42.42.0/24 -o ens192 -j MASQUERADE
COMMIT
# Completed on Fri Apr 15 18:03:58 2016
[root@localhost strongswan]#

I originally started out using the new firewall-cmd in Centos 7 using the following commands, but I ran into the exact same problem, hoping that moving back to old iptables would help since I honestly understand a bit better. Here were the commands I ran back then:

firewall-cmd --zone=dmz --permanent --add-rich-rule='rule protocol value="esp" accept' # ESP (the encrypted data packets)
firewall-cmd --zone=dmz --permanent --add-rich-rule='rule protocol value="ah" accept' # AH (authenticated headers)
firewall-cmd --zone=dmz --permanent --add-port=500/udp #IKE  (security associations)
firewall-cmd --zone=dmz --permanent --add-port=4500/udp # IKE NAT Traversal (IPsec between natted devices)
firewall-cmd --permanent --add-service="ipsec"
firewall-cmd --zone=dmz --permanent --add-masquerade
firewall-cmd --set-default-zone=dmz
firewall-cmd --reload
firewall-cmd --list-all

Any ideas on what I'm doing wrong? Why does the ping request/response make it all the way to the target and back to the StrongSwan server but not back to the client?

That there are no outbound cleartext packets seen in tcpdump [is normal](https://wiki.strongswan.org/projects/strongswan/wiki/FAQ#Capturing-outbound-plaintext-packets-with-tcpdumpwireshark). You should probably check if there are ESP packets leaving the host or if the traffic counters increase (in `ipsec statusall` or `ip -s xfrm state`). Also, have a look at [CorrectTrafficDump](https://wiki.strongswan.org/projects/strongswan/wiki/CorrectTrafficDump) on the strongSwan wiki and the TRACE target in iptables to see what's going on in the firewall. — ecdsa, Apr 22 '16 at 13:19
@ecdsa First, I want you to know that you rock. All the best strongswan related answers on this site come from you. You deserve way more props. — Ryan Shillington, Apr 22 '16 at 17:16
Second, this response really helped me. I set up iptables & tcpdump to monitor egress ESP and I can see the packets flowing back to the client. I tried pinging with 256 bytes and I see the packets fluctuating by an extra 256 bytes so strongswan is definitely trying to send the data back. I set up wireshark on my windows machine and I only see outgoing ESP packets. Is it a similar problem on the windows side that wireshark doesn't show me incoming ESP packets or are they really being dropped somewhere? — Ryan Shillington, Apr 22 '16 at 17:18
Another hint is that `ip xfrm policy` show the correct policy when the client connects. But after a few mins the same command shows only the 0.0.0.0 policies (my client ones are missing). The windows client still shows the VPN as connected. Then when I ping again, the xfrm policy gets added back. Maybe that's expected behavior? — Ryan Shillington, Apr 22 '16 at 17:22
Oh, Also, I see ISAKMP packets being sent and received. Just not ESP packets being received. — Ryan Shillington, Apr 22 '16 at 17:27

Ryan Shillington · Answer 1 · 2016-04-22T18:57:40.233

Well, I got this to work by giving my Strongswan machine a public IP. That was the only change needed. So now my network looks like:

[Windows 2012 Client      ]------[External Nat]-----[VPN         ]---------[Hidden network]
10.185.28.241 / 10.42.42.1        x.x.x.71           10.112.18.105           10.112.0.0/16

That's all that's required. I didn't have to change any of my configs. I just changed the DNS name that was pointing to 10.112.18.105 to x.x.x.71 and left everything else alone.

To the best of my understanding, the AWS network where my Windows client is was somehow eating the returning ESP packets so it never got a response from the server when going through the site to site VPN. I didn't have to make any changes at all on my Strongswan server. Just going over a public IP and NOT going through the site to site VPN was all I needed.

This might be due to AWS's source/destination check (as noted on [our wiki](https://wiki.strongswan.org/projects/strongswan/wiki/AwsVpc)). — ecdsa, Apr 23 '16 at 05:44
It's disabled (and always has been), which I think is the recommendation. — Ryan Shillington, Apr 25 '16 at 21:25

How do i get Strongswan / IPTables to route data back to my road warrior client correctly?

ipsec.conf:

Strongswan statusall:

Network Information

IPTables

1 Answers1