Force Windows IPSec/L2TP VPN to use AES in IPSec main mode

Question

The RAS-based IPSec VPN client in Windows does not seem to respect the IPSec defaults in Windows Firewall (which hosts the IPSec driver), but insists on using 3DES encryption with SHA1 integrity for key exchange (a.k.a. IPSec main mode). Both of these are legacy algorithms now.

So it doesn't matter what encryption is adopted for data transfer (quick mode), which RAS supports up to AES-256-CBC, the whole link is only as weak as 3DES/SHA1.

On the other hand, the IPSec driver in Windows Firewall can handle SHA-384, AES-GCM and ECDH P-384, so is there a way to configure the VPN client to use these or simply respect the defaults set in Windows Firewall?

Answer 1

Have similar problem with IPSec/IKE. It seems that RasMan service completely disrespects IPSec policies configured via Windows Firewall. And the best I was able to came up with is AES-SHA1-DH2048 through registry mangling. I stored it as .reg file, there's comments so things should be pretty clear.

REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\IKEv2]
"CustomParams"=dword:00000001
"CustomProposalsCount"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\IKEv2\Proposals]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\IKEv2\Proposals\0]
; for "Quick Mode", all keys optional
; DES, 3DES, AES_128, AES_256
"esp_encr"="AES_128"
; MD5, SHA1
"esp_auth"="SHA1"
; MD5, SHA1
;"AH"="SHA1"
; NONE, 1, 2, 2048, ECP_256, ECP_384, MM
;"PFS"="MM"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\Parameters]
"AllowL2TPWeakCrypto"=dword:00000000
"AllowPPTPWeakCrypto"=dword:00000000
; for "Main Mode"
; 0 - disable, 1 - enable, 2 - force /// WARNING! "force" disables stronger DH groups!
"NegotiateDH2048_AES256"=dword:00000001