Why do IPSec VPN Compression and Windows Remote Desktop not work together?

2

I just discovered a strange problem. I connect to a remote network via VPN. Within this VPN is a separate Microsoft Small Business Server (which does not provide the VPN, another device does this). So the setup is:

Windows with third party VPN Client <-> (WAN) <-> VPN Gateway <-> SBS

I can ping the SBS and I can access the shared folders just perfectly fine when connected to the remote network via VPN. Also all other programs I use work just fine. This works with IPSec compression activated and deactived as well, in both cases without problems.

The only thing that does not work if having IPSec compression enabled is the remote desktop connection between my Windows client and the SBS. It works as it should, however, if the IPSec compression is disabled. The only error I get from the RDC in this case is the nonsense message "An internal error has occured".

How can the SBS remote desktop fail (but everything else works fine) if the compression/decompression is handled between the VPN Gateway and my VPN client? Shouldn't this be one "layer" deeper? The Remote Desktop application should not even know that the data was compressed and decompressed, should it?

Is there any way to have RDC and IPSec compression work together?

Foo Bar

Posted 2014-10-21T19:43:34.953

Reputation: 1 270

Can you please telnet from the client to the server on port 3389 (RDP port) and see what happens? Network traces might help a bit more as well (Microsoft Message Analyzer or netsh will help with this). – cdavid – 2014-10-21T22:04:59.317

@cdavid Nothing happens (with IPsec compression disabled). Connection can not be established. Telnet on Port 23 works however. Could it be some MTU mismatch problem? – Foo Bar – 2014-10-30T19:02:22.690

No answers