strongSwan IKEv2 + Windows 7 Agile VPN: What is causing Error 13801

Question

I have an AWS instance that I want to be a VPN server. It will connect Windows 7 clients to a private network in the Amazon cloud.

• I have installed Ubuntu 12.04 and the strongswan-ikev2 package.
• ipsec version reports Linux strongSwan U4.5.2/K3.2.0-52-virtual
• Note that both the client and server are behind NAT (the client because it is on a local office network, and the server because it is in Amazon's cloud). I have unblocked UDP ports 500 and 4500 on both the Amazon dashboard and the client's firewall.

• This is /etc/ipsec.conf:

  config setup
      plutostart=no
  
  conn %default
      keyexchange=ikev2
      ike=aes256-sha1-modp1024!
      esp=aes256-sha1!
      dpdaction=clear
      dpddelay=300s
      rekey=no
  
  conn win7vpn
      left=%any
      leftsubnet=<amazon VPC CIDR block>
      leftauth=pubkey
      leftcert=openssl-cert.pem
      leftid=<vpn server public dns name>
      right=%any
      rightsourceip=<amazon private IP address, which elastic ip is forwarded to>
      rightauth=eap-mschapv2
      rightsendcert=never
      eap_identity=%any
      auto=add
  

• This is /etc/ipsec.secrets:

  : RSA openssl-key.rsa
  TESTDOMAIN\testuser : EAP "testpassword"
  

• I have added the CA certificate that signed the server's host cert to the local machine (not user) cert store so that Windows can authenticate the server.

I then try to connect to the server using the Windows 7 client as prescribed here, with one exception - I'm using the DNS name rather than the IP address. I enter the username, domain, and password in my ipsec.secrets file, and it tries to connect.

When it does, I get strongSwan logs that look like this. I munged these a bit for both censorship and clarity; CLIENTPUB/CLIENTPRIV are the client's public and private IP addresses and AMAZONPRIV is the server's private IP address (which is what the server's public IP - Amazon calls this an "Elastic IP" - forwards to).

Sep  4 00:16:17 localhost charon: 14[IKE] CLIENTPUB is initiating an IKE_SA
Sep  4 00:16:17 localhost charon: 14[NET] received packet: from CLIENTPUB[500] to AMAZONPRIV[500]
Sep  4 00:16:17 localhost charon: 14[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) ]
Sep  4 00:16:17 localhost charon: 14[IKE] CLIENTPUB is initiating an IKE_SA
Sep  4 00:16:17 localhost charon: 14[IKE] local host is behind NAT, sending keep alives
Sep  4 00:16:17 localhost charon: 14[IKE] remote host is behind NAT
Sep  4 00:16:17 localhost charon: 14[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(MULT_AUTH) ]
Sep  4 00:16:17 localhost charon: 14[NET] sending packet: from AMAZONPRIV[500] to CLIENTPUB[500]
Sep  4 00:16:17 localhost charon: 15[NET] received packet: from CLIENTPUB[4500] to AMAZONPRIV[4500]
Sep  4 00:16:17 localhost charon: 15[ENC] unknown attribute type INTERNAL_IP4_SERVER
Sep  4 00:16:17 localhost charon: 15[ENC] parsed IKE_AUTH request 1 [ IDi CERTREQ N(MOBIKE_SUP) CP(ADDR DNS NBNS SRV) SA TSi TSr ]
Sep  4 00:16:17 localhost charon: 15[IKE] received cert request for "C=US, ST=TX, O=Test CA, CN=Test CA"
Sep  4 00:16:17 localhost charon: 15[IKE] received 316 cert requests for an unknown ca
Sep  4 00:16:17 localhost charon: 15[CFG] looking for peer configs matching AMAZONPRIV[%any]...CLIENTPUB[CLIENTPRIV]
Sep  4 00:16:17 localhost charon: 15[CFG] selected peer config 'dlpvpn'
Sep  4 00:16:17 localhost charon: 15[IKE] initiating EAP-Identity request
Sep  4 00:16:17 localhost charon: 15[IKE] peer supports MOBIKE
Sep  4 00:16:17 localhost charon: 15[IKE] authentication of 'C=US, ST=TX, O=DLP Test CA, CN=vpn.example.com' (myself) with RSA signature successful
Sep  4 00:16:17 localhost charon: 15[IKE] sending end entity cert "C=US, ST=TX, O=DLP Test CA, CN=vpn.example.com"
Sep  4 00:16:17 localhost charon: 15[ENC] generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ]
Sep  4 00:16:17 localhost charon: 15[NET] sending packet: from AMAZONPRIV[4500] to CLIENTPUB[4500]

At this point, Windows pops up an error message immediately:

Verifying user name and password...
Error 13801: IKE authentication credentials are unacceptable

After a few seconds, charon tries again and then closes the connection.

Sep  4 00:16:37 localhost charon: 16[IKE] sending keep alive
Sep  4 00:16:37 localhost charon: 16[NET] sending packet: from AMAZONPRIV[4500] to CLIENTPUB[4500]
Sep  4 00:16:47 localhost charon: 03[JOB] deleting half open IKE_SA after timeout

And that's it.

As far as I can tell, I'm following all of the instructions on the strongSwan wiki.

What am I doing wrong here?

Edit: this is definitely a problem with certificates. I disabled the extended validation checks by editing the registry and rebooting as described in MSKB926182 (lol if you wanted a link to that) and I can now connect to my VPN server with no errors. I'll figure out how to generate certs that satisfy the requirements and add an answer. Thanks to @ecdsa for the pointer to the cert page on the strongSwan wiki that got me pointed in the right direction.

Answer 1

Figured this out. @ecdsa pointed me in the right direction, and I finally was able to solve the problem by following this guide.

ipsec pki --gen --type rsa --size 4096 --outform pem > vpnca.key.pem
ipsec pki --self --flag serverAuth --in vpnca.key.pem --type rsa --digest sha1 \
    --dn "C=US, O=Example Company, CN=Example VPN CA" --ca > vpnca.crt.der
ipsec pki --gen --type rsa --size 4096 --outform pem > vpn.example.com.key.pem
ipsec pki --pub --in vpn.example.com.key.pem --type rsa > vpn.example.com.csr
ipsec pki --issue --cacert vpnca.crt.der --cakey vpnca.key.pem --digest sha1 \
    --dn "C=US, O=Example Company, CN=vpn.example.com" \
    --san "vpn.example.com" --flag serverAuth --outform pem \
    < vpn.example.com.csr > vpn.example.com.crt.pem 
openssl rsa -in vpn.example.com.key.pem -out vpn.example.com.key.der -outform DER

cp vpnca.crt.der /etc/ipsec.d/cacerts
cp vpn.example.com.crt.pem /etc/ipsec.d/certs
cp vpn.example.com.key.der /etc/ipsec.d/private

About the error

The error message was "Error 13801: IKE authentication credentials are unacceptable", which sounded like my user credentials weren't working. However, this is a message about authenticating the server, which is done (per my configuration) by the server's SSL certificate. Microsoft has published documentation on Troubleshooting IKEv2 VPN Connections that lists possible causes for this error:

• The certificate is expired.
• The trusted root for the certificate is not present on the client.
• The subject name of the certificate does not match the remote computer.
• The certificate does not have the required Enhanced Key Usage (EKU) values assigned.

In my case, my problem had to do with the EKU values. Following the guide I linked at the top, I was able to generate a certificate with the correct EKU values, and it worked great.

To troubleshoot this, you can disable EKU checking on your Windows client (of course, this should only be done for testing):

• Launch regedit
• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\RasMan\Parameters
• Add a DWORD called DisableIKENameEkuCheck, and set its value to 1
• Microsoft documentation instructs you to reboot after doing this, but I didn't need to in order for this to take effect.

Answer 2

I had an identical problem and solved it by ensuring I had the certificate chain in the certificate file (end-entity cert, intermediate CA, root CA - in that order). TLS is fun.

After restarting strongSwan, this stopped working, but started working again when I dropped the intermediate and root CA into /etc/ipsec.d/cacerts.

Answer 3

After a long search, this thread got my Windows Phone 10 (WP10) configuration working with IKEv2! One thing to mention might be that you have to ./configure your Strongswan with --enable-eap-identity --enable-eap-mschapv2 --enable-openssl (and probably --enable-dhcp) to have the necessary plugins. And yes, you need to get the certs right (on the server side - the client only needs to know the root CA of the server).