UDP-Packets seem to get lost in IPsec tunnel from Strongswan to AWS cloud - connection works with Openswan

Question

Use case: IOT-device connected through AWS cloud

The IOT-device is behind a router that sends all traffic through aws cloud.

The IOT-server can not be configured and thus is not part of the AWS cloud

For configuration, the IOT-device needs to be notfied with an UPD packet to port xxxxx to establish an administion connection. This udp packet can not be sent directly to AWS cloud

Thus, we need a communication server to route the UDP packet:

Routing on the IOT-server can not be configured, thus, the UDP packet needs to be send to zz.zz.zz.zz

The communication server runs debian 10 with strongswan

ipsec.conf:

conn %default
    mobike=no
    compress=no
    authby=secret
    keyexchange=ike
    ike=aes128-sha1-modp1024!
    ikelifetime=8h
    esp=aes128-sha1-modp1024!
    lifetime=1h
    rekeymargin=3m
    keyingtries=%forever
    installpolicy=yes
    dpdaction=restart
    type=tunnel

conn dc-aws1
    leftsubnet=zz.zz.zz.zz #local subnet
    right=vv.vv.vv.vv # AWS Gateway Public IP
    rightsubnet=xx.xx.0.0/16 #remoye subnet
    auto=start


include /var/lib/strongswan/ipsec.conf.inc

The following parts of the connection work: Standard operation works just fine.

The ipsec connection is running (as expected):

sudo ipsec status
Security Associations (1 up, 0 connecting):
dc-aws1[3]: ESTABLISHED 11 seconds ago, zz.zz.zz.zz[zz.zz.zz.zz]...vv.vv.vv.vv[vv.vv.vv.vv]
dc-aws1{16}: INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: cd6dfea5_i 401dc4d5_o
dc-aws1{16}: zz.zz.zz.zz/32 = xx.xx.0.0/16
    dc-aws1{17}: INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: c2507a98_i 9d083aa4_o
    dc-aws1{17}:  zz.zz.zz.zz/32 = xx.xx.0.0/16


sudo ip xfrm policy show

src zz.zz.zz.zz/32 dst xx.xx.0.0/16
dir out priority 375423 ptype main
tmpl src zz.zz.zz.zz dst vv.vv.vv.vv
proto esp spi 0x9d083aa4 reqid 2 mode tunnel
src xx.xx.0.0/16 dst zz.zz.zz.zz/32
dir fwd priority 375423 ptype main
tmpl src vv.vv.vv.vv dst zz.zz.zz.zz
proto esp reqid 2 mode tunnel
src xx.xx.0.0/16 dst zz.zz.zz.zz/32
dir in priority 375423 ptype main
tmpl src vv.vv.vv.vv dst zz.zz.zz.zz
proto esp reqid 2 mode tunnel

Ping works between the router and the communication server though the vpn connection.

Traceroute also works if icmp packets are used.

In order to forward the upd packet to the IOT-device, network address translation is used with iptables

iptables -t nat -I PREROUTING -p udp -s yy.yy.yy.yy --dport xxxxx -j DNAT --to xx.xx.xx.xx

xfrm policy does not apply, if the source is yy.yy.yy.yy, thus source network address translation is used as well

iptables -t nat -I POSTROUTING -p udp -s yy.yy.yy.yy --dport xxxxx -j SNAT --to-source zz.zz.zz.zz

also a forwarding rule is needed

iptables -I FORWARD -p udp -d xx.xx.xx.xx --dport xxxxx -j ACCEPT

tcpdump shows, that udp packets arrive and are forwarded (in between, there are alive messages for the vpn connection):

sudo tcpdump -n -i any host vv.vv.vv.vv or port xxxxx
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:22:48.520734 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: NONESP-encap: isakmp: child_sa inf2[I]
08:22:48.535700 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: NONESP-encap: isakmp: child_sa inf2[R]
08:22:56.717778 IP yy.yy.yy.yy.54278 > zz.zz.zz.zz.xxxxx: UDP, length 108
08:22:56.717908 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x81f1d489,seq=0x1), length 180
08:23:06.344622 IP yy.yy.yy.yy.46955 > zz.zz.zz.zz.xxxxx: UDP, length 108
08:23:06.344749 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x81f1d489,seq=0x2), length 180
08:23:10.797048 IP yy.yy.yy.yy.33667 > zz.zz.zz.zz.xxxxx: UDP, length 108
08:23:10.797247 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x81f1d489,seq=0x3), length 180
08:23:18.521104 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: NONESP-encap: isakmp: child_sa inf2[I]
08:23:18.536895 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: NONESP-encap: isakmp: child_sa inf2[R]
08:23:25.423142 IP yy.yy.yy.yy.40703 > zz.zz.zz.zz.xxxxx: UDP, length 108
08:23:25.423271 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x81f1d489,seq=0x4), length 180
08:23:31.756269 IP yy.yy.yy.yy.58584 > zz.zz.zz.zz.xxxxx: UDP, length 108
08:23:31.756378 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x81f1d489,seq=0x5), length 180
^C
14 packets captured
14 packets received by filter
0 packets dropped by kernel

What does not work:

However the udp pakets seem to get lost. In the logs at aws, no traffic is visible in the tunnel. Also, no packets arrive at the router.

Traceroute with udp and tcp packets does not work.

The problem can be reproduced when running netcat on the communication server in listening mode and connecting to it from behind the router. In the tcp dump, the syn packets are arriving, a response seems to be sent out, but there is no traffic arriving from the communication server in the aws cloud. tcpdump from the communication server for this test:

11:35:06.597736 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: UDP-encap: ESP(spi=0xcb99370a,seq=0x1), length 100
11:35:06.597736 IP xx.xx.xx.xx.49768 > zz.zz.zz.zz.15952: Flags [S], seq 101710370, win 64240, options [mss 1350,sackOK,TS val 2355221232 ecr 0,nop,wscale 7], length 0
11:35:06.598157 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0xb), length 100
11:35:07.534252 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: UDP-encap: ESP(spi=0xcb99370a,seq=0x2), length 100
11:35:07.534252 IP xx.xx.xx.xx.49768 > zz.zz.zz.zz.15952: Flags [S], seq 101710370, win 64240, options [mss 1350,sackOK,TS val 2355222233 ecr 0,nop,wscale 7], length 0
11:35:07.534445 IP zz.zz.zz.zz.4500 > vv.vv.vsv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0xc), length 100
11:35:08.561060 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0xd), length 100
11:35:09.559712 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: UDP-encap: ESP(spi=0xcb99370a,seq=0x3), length 100
11:35:09.559712 IP xx.xx.xx.xx.49768 > zz.zz.zz.zz.15952: Flags [S], seq 101710370, win 64240, options [mss 1350,sackOK,TS val 2355224249 ecr 0,nop,wscale 7], length 0
11:35:09.559908 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0xe), length 100
11:35:11.569079 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0xf), length 100
11:35:13.672232 IP vv.vv.vv.vv.4500 > zz.zz.zz.zz.4500: UDP-encap: ESP(spi=0xcb99370a,seq=0x4), length 100
11:35:13.672232 IP xx.xx.xx.xx.49768 > zz.zz.zz.zz.15952: Flags [S], seq 101710370, win 64240, options [mss 1350,sackOK,TS val 2355228377 ecr 0,nop,wscale 7], length 0
11:35:13.672319 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0x10), length 100
11:35:17.713025 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0x11), length 100
11:35:25.905124 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0x12), length 100
11:35:42.033153 IP zz.zz.zz.zz.4500 > vv.vv.vv.vv.4500: UDP-encap: ESP(spi=0x9a2c6938,seq=0x13), length

It is not clear to me, where the packets could get lost. Any hint on how to narrow down the problem is welcome

** Update **

In the meanwhile I double-checked configuration, but with no success.

Then, I switched to Openswan (2.6.51.5), which is tested by AWS.

With Openswan packets arrive in the cloud as expected.

My conclusion is, that Strongswan is incompatible with AWS VPC.

I'm happy to make further tests if there are ideas on how to test further.

I can't understand your environment configuration. Can you draw a diagram including IPs, protocols, servers, etc? Network level diagnostics in AWS are done with VPC Flow Logs, turning them on and watching them will be useful. They're about as user friendly as tcpdump, but read the documentation and they're ok. — Tim, Aug 08 '20 at 08:11
I had a similar issue once with ipsec ... These tunnels are ugly to debug! My issue was that I had a typo inside the tunnel parameters (especially inside the psk). The tunnel came up despite the typo, but we also observed packetloss. My advice is to double-check all your tunnel configuration! — Martin, Aug 12 '20 at 15:19

score 0 · Accepted Answer · answered Oct 21 '20 at 11:55

Thanks to Doug Tucker on the Strongswan User List I was able to get the connection running with Strongswan as well.

My best guess is, that the problem seems related to the xfrm policies.

The following setup (ipsec.conf) works for me:

config setup
    # strictcrlpolicy=yes
    uniqueids = no

# Add connections here.
conn %default
#   mobike=no
    compress=no
    authby=secret
    keyexchange=ike
    ike=aes128-sha1-modp1024!
    ikelifetime=8h
    esp=aes128-sha1-modp1024!
    lifetime=1h
#   rekeymargin=3m
    keyingtries=%forever
#   installpolicy=yes
    type=tunnel
        leftauth=psk
        rightauth=psk
        dpddelay=10s
        dpdtimeout=30s
    dpdaction=restart



conn dc-aws1
    auto=start
        left=zx.zx.zx.zx
        leftid=zz.zz.zz.zz
        leftsubnet=zx.zx.zx.zx/24
    right=vv.vv.vv.vv # AWS Gateway Public IP
    rightsubnet=xx.xx.0.0/16 #remoye subnet
        mark=100
        leftupdown="/usr/local/bin/aws-updown.sh -ln AWSTunnel1 -ll 169.254.6.2/30 -lr 169.254.6.1/30 -m 100 -r xx.xx.0.0/16"

Side note: The vpn-connection was moved to an virtual machine. zx.zx.zx.zx is the IP address of the virtual machine where the vpn connection now runs (hosted on zz.zz.zz.zz).

aws-updown.sh was also provided on the strongswan user list, but can also be found on github:

#!/bin/bash

while [[ $# > 1 ]]; do
        case ${1} in
                -ln|--link-name)
                        TUNNEL_NAME="${2}"
                        TUNNEL_PHY_INTERFACE="${PLUTO_INTERFACE}"
                        shift
                        ;;
                -ll|--link-local)
                        TUNNEL_LOCAL_ADDRESS="${2}"
                        TUNNEL_LOCAL_ENDPOINT="${PLUTO_ME}"
                        shift
                        ;;
                -lr|--link-remote)
                        TUNNEL_REMOTE_ADDRESS="${2}"
                        TUNNEL_REMOTE_ENDPOINT="${PLUTO_PEER}"
                        shift
                        ;;
                -m|--mark)
                        TUNNEL_MARK="${2}"
                        shift
                        ;;
                -r|--static-route)
                        TUNNEL_STATIC_ROUTE="${2}"
                        shift
                        ;;
                *)
                        echo "${0}: Unknown argument \"${1}\"" >&2
                        ;;
        esac
        shift
done

command_exists() {
        type "$1" >&2 2>&2
}

create_interface() {
        ip link add ${TUNNEL_NAME} type vti local ${TUNNEL_LOCAL_ENDPOINT} remote ${TUNNEL_REMOTE_ENDPOINT} key ${TUNNEL_MARK}
        ip addr add ${TUNNEL_LOCAL_ADDRESS} remote ${TUNNEL_REMOTE_ADDRESS} dev ${TUNNEL_NAME}
        ip link set ${TUNNEL_NAME} up mtu 1419
}

configure_sysctl() {
        sysctl -w net.ipv4.ip_forward=1
        sysctl -w net.ipv4.conf.${TUNNEL_NAME}.rp_filter=2
        sysctl -w net.ipv4.conf.${TUNNEL_NAME}.disable_policy=1
        sysctl -w net.ipv4.conf.${TUNNEL_PHY_INTERFACE}.disable_xfrm=1
        sysctl -w net.ipv4.conf.${TUNNEL_PHY_INTERFACE}.disable_policy=1
}

add_route() {
        IFS=',' read -ra route <<< "${TUNNEL_STATIC_ROUTE}"
        for i in "${route[@]}"; do
            ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}
        done
        iptables -t mangle -A FORWARD -o ${TUNNEL_NAME} -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
        iptables -t mangle -A INPUT -p esp -s ${TUNNEL_REMOTE_ENDPOINT} -d ${TUNNEL_LOCAL_ENDPOINT} -j MARK --set-xmark ${TUNNEL_MARK}
        ip route flush table 220
}

cleanup() {
        IFS=',' read -ra route <<< "${TUNNEL_STATIC_ROUTE}"
        for i in "${route[@]}"; do
            ip route del ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}
        done
        iptables -t mangle -D FORWARD -o ${TUNNEL_NAME} -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
        iptables -t mangle -D INPUT -p esp -s ${TUNNEL_REMOTE_ENDPOINT} -d ${TUNNEL_LOCAL_ENDPOINT} -j MARK --set-xmark ${TUNNEL_MARK}
        ip route flush cache
}

delete_interface() {
        ip link set ${TUNNEL_NAME} down
        ip link del ${TUNNEL_NAME}
}

# main execution starts here

command_exists ip || echo "ERROR: ip command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2
command_exists iptables || echo "ERROR: iptables command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2
command_exists sysctl || echo "ERROR: sysctl command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2

case "${PLUTO_VERB}" in
        up-client)
                create_interface
                configure_sysctl
                add_route
                ;;
        down-client)
                cleanup
                delete_interface
                ;;
esac

UDP-Packets seem to get lost in IPsec tunnel from Strongswan to AWS cloud - connection works with Openswan

1 Answers1