27

Many admins keep perpetuating - on ServerFault and elsewhere - how bad of an idea TCP-over-TCP is, e.g. in VPNs. That even the slightest packet loss will make one suffer from at least severe throughput degradation if not TCP meltdown, and that TCP-over-TCP is therefore strictly to be avoided. And that probably was once all true, e.g. 2001 when this article was written that is still referred to.

But since then we've seen major advances in technology and protocols. Nowadays we have 'Selective ACK' implemented almost everywhere, and Moore's law has given us so much more memory, and with it came large TCP buffers optimized for Gbit uplinks. Also packet loss is much less of an issue these days on non-radio links. All this should alleviate the TCP-over-TCP problem significantly, shouldn't it?

Note that there are real-world scenarios where e.g. TCP-based VPNs are easier to implement and operate than UDP/ESP-based ones (see more below). Therefore my question:

Under what circumstances (link packet loss & latency) is TCP-over-TCP performing significantly worse that TCP alone, assuming SACK support and decently sized TCP buffers on both ends?

It would be great so see some measurements that show the correlations between (outer connection) packet loss/latency, and (inner connection) throughput/jitter - for TCP-over-TCP, and for TCP alone. I found this interesting article, but it seems to be concerned about latency only, and to not address (outer) packet loss.

Also: Are there recommended settings (e.g. TCP options, buffer settings, reducing MTU/MSS, etc) to narrow the performance gap between TCP and TCP-over-TCP?

Update: Our rationale.

This question is still very relevant in some real-world scenarios. E.g. we deploy embedded devices in large buildings that collect sensor data and feed it into our platform via VPN. The problem we are facing is firewalls and improperly configured uplinks that our not under our control, combined with reluctant IT departments. See a detailed example discussed here.

In a lot of such cases, switching from non-TCP to a TCP-based VPN (very easy if you use OpenVPN like us) is a quick fix that allows us to evade uphill finger-pointing battles. E.g. often TCP port 443 is generally allowed (at least via proxy), or we can overcome Path-MTU issues by simply reducing TCP's MSS option.

It would be good to know under what circumstances a TCP-based VPN can be considered a viable alternative, so we can make an informed decision outweighing the pros and cons of either option. For example we know that TCP-VPN is ok for us on non-radio links, but we do have a fair share of remote clients on 3G uplinks with significant packet loss and high latency - how would a TCP-VPN perform there?

I tried to improve the title and the central question accordingly; i hope it makes sense.

Community
  • 1
Nils Toedtmann
  • 3,202
  • 5
  • 25
  • 36
  • You will quickly notice the difference between TCP over TCP vs UDP (etc) over TCP VPNs when using them for interactive sessions, e.g. ssh. You'll notice latency if not session drops. YMMV, TIAS – Daniel S. Sterling Dec 07 '15 at 20:17
  • Only if the 'outer' connection is subject to a certain degree of latency or packet loss in the first place. I have plenty of SSH sessions over TCP-based VPNs, and many work just fine with no noticeable lag. – Nils Toedtmann Dec 08 '15 at 15:59
  • Indeed -- it works when you have a good network. If you don't always have a good network, it doesn't always work – Daniel S. Sterling Dec 09 '15 at 16:24
  • What is a good network? If everything's running in a single AWS region, is the network good enough? – rich remer Nov 07 '18 at 20:49

1 Answers1

7

I think it's actually more debated than you make it appear. There is an admittedly old, related Linux FAQ: http://www.tldp.org/HOWTO/VPN-HOWTO/

I've used a PPP-over-ssh-over-ADSL for more than 12 years, and it never failed me, so from my experience I'd dare to say that the doomsayers probably largely exaggerate. TCP over TCP is probably a bad idea with RTC connections, satellite links and other links with either very low throughput or very long delays, but for most uses it just works.

Now the true question is: why would you use TCP over TCP at all? When I've set up my PPP-over-ssh, it was largely because back then it was by far the easiest way to build a quick VPN; then I've used it ever since out of sheer laziness.

Nowadays there are practical, easy to set up tools like OpenVPN, IPSec VPNs so that you shouldn't ever need to bother you with this TCP-over-TCP problem.

wazoox
  • 6,782
  • 4
  • 30
  • 62
  • 1
    There are situations where TCP-over-TCP is a simple fix to a number of networking issues. I added a section elaborating our rationale. – Nils Toedtmann Sep 26 '14 at 08:57
  • I hope for an answer that is more specific about the conditions under which TCP-over-TCP becomes a problem. One of our use cases *is* radio links that have varying degrees of latency and packet loss. – Nils Toedtmann Sep 29 '14 at 18:39
  • TCP over TCP over TCP (TCP traffic in a TCP OpenVPN accessed through a TCP SSH forward) cost me about 5Mb/s. It never failed me but I wouldn't recommend it just because it's a huge waste. – Allison May 30 '17 at 21:12