11

I know just enough networking to be dangerous. The nitty gritty low level details of NAT are not something I am particularly knowledgeable about.

I accidentally found myself in a discussion earlier today about placing a bunch of our nodes behind a NAT Gateway. (1 public IP address and X private LAN addresses). I called up the 16 bit limit to source and destination port fields in the TCP protocol, (http://www.ietf.org/rfc/rfc793.txt - page 15) and mentioned that it would limit us to some 65,000 connections (65536). -- I am not so confident about that answer anymore. Can you help me with some details?

I understand that an incoming port (server port) on our side can accept as many connections as there are sourceIP x SourcePort combinations. Let's discount those for the time being and focus on connections originating in the LAN, traveling through the NAT Gateway, and ending on a random host at a random port.

On a normal [Linux] system, outgoing connections I believe are limited to 1 per port per Source IP. If we pretend that we live in a simple world where each system only has 1 IP address, then a 'normal system' would be limited to an absolute maximum of 65536 connections.

1) In TCP is a single source IP limited to 65536 MAX theoretical outgoing connections?

2) Or is the limit actually 65536 connections for each Remote Host?

2) [Written another way]: Can the same source port be used for a different remoteHostIP:RemotePort combination?

For example: (Is the following OK?)

Source IP   |Source Port |Remote IP|Remote Port   
192.168.0.20:36500   -->    8.8.8.8:23
192.168.0.20:36500   -->    8.8.4.4:23

3) Are the answers to questions 1 and 2 different for a ...'not normal system' [Cisco router acting as a NAT Gateway]?

Ex: A specialized networking device that has one public facing IP and up to ~65,000 Lan IPs [or more] behind it? Is there magic at place or is the answer to question 2 just always: yes? (or no)

4) The above questions all assume a stateful TCP connection. Is the story any different with a stateless conection like UDP?

And Ultimately:

5) Will our LAN be limited to 65536 (or some other theoretical limit) concurrent connections to the outside world through a single public IP address?

Thank you! :)

For purposes of this question, we are behind very BEEFY AND BRAND NEW Cisco Nexus gear (7000 series I think). It may be better to ignore memory/etc limitations unless they can be specifically quantified.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Daniel Widrick
  • 3,418
  • 2
  • 12
  • 26
  • http://serverfault.com/questions/533611/how-do-high-traffic-sites-service-more-than-65535-tcp-connections - Deals with incoming connections [A server serving x unique clients on its own server port]... I am asking specifically about outgoing connections through a NAT Gateway. http://socpuppet.blogspot.com/2013/03/avoiding-tcpudp-port-exhaustion-cisco.html -- That 'blog post' suggests that port exhaustion is a real problem when there are many hosts behind a gateway and suggests "NAT Groups" and "NAT POOLS" -- breaking the LAN up and placing behind separate public IPs as a solution? -researching – Daniel Widrick Sep 26 '13 at 00:04

5 Answers5

9

Correct me if I'm wrong but this is the way I understand it. The limits are per client / server / port. So in light of that.

1) In TCP is a single source IP limited to 65536 MAX theoretical outgoing connections?

No. I believe it's limited to 65536 theoretical max to the same destination IP.

Windows workstations (non server versions) have limits imposed which make this number much less. Linux has resource limits, but they generally aren't hit by the average user and you can easily tweak them.

You'll probably hit other resource limits as you start increasing the number anywhere near 64K.

Consumer grade routers might have much lower limits due to the limited resources.

2) Or is the limit actually 65536 connections for each Remote Host?

Yes

3) Are the answers to questions 1 and 2 different for a ...'not normal system' [Cisco router acting as a NAT Gateway]?

No

4) The above questions all assume a stateful TCP connection. Is the story any different with a stateless conection like UDP?

UDP is connectionless. So this isn't really relevant to UDP.

5) Will our LAN be limited to 65536 (or some other theoretical limit) concurrent connections to the outside world through a single public IP address?

No.

In the context of stateful firewalls that track connections and provide other tracking features, yes these modules themselves may have limits. The op has not said anything about which firewall/NAT router is being used so we can't even speculate as to what limitations it might impose at the moment.

hookenz
  • 14,132
  • 22
  • 86
  • 142
  • 4
    UDP is affected by the same limits. When a UDP packet leaves a NAT firewall it has to hold that port and remote ip address open for a certain amount of time for return packets to make it through. – longneck Sep 26 '13 at 02:05
  • 'Other resource limits' are not particularly interesting in our discussion. I suspect you may be correct in some (or all) of your answers. Can you provide any supporting documentation? – Daniel Widrick Sep 26 '13 at 02:25
  • 1
    File handles, memory, cpu overhead. You can have lots and lots of active, open connections but there is overhead and what overhead varies between operating systems. – hookenz Sep 26 '13 at 03:34
  • @longneck - you're right in the context of NAT of course. Especially for stateful firewalls. – hookenz Sep 26 '13 at 03:36
  • 3
    As per the Socket Reuse comments on user8162's answer, It is conceivable to build a NAT platform that uses the full 5 part tuple (protocol [udp/tcp], source ip, destination ip, source port, destination port) as a unique key for each outgoing socket. Host1:port1 -> host2:port2 and Host1:port1 -> host3:port2 are conceivable layouts that could in theory point to separate sockets. With that in mind, until I see something in the TCP spec or other official document that prevents this functionality, I will have to accept Matt's answer as correct. (taking into account the udp considerations). – Daniel Widrick Sep 26 '13 at 05:43
  • 1
    @lVlint67 in fact that's the only conceivable way to uniquely key each NAT'd connection. Anything less than that would lead to unexpectedly blocked connections. (You could argue the protocol not needing to be a part of the key, but why should an outgoing TCP connection allow UDP packets to return?) – longneck Sep 26 '13 at 15:36
2

Because there are many approaches to NAT design, the answer of course is "it depends." Geoff Huston wrote an excellent overview of various NAT designs.

In my experience many small-office/home-office routers will not handle > 64k addresses to the same address, but not just because of port exhaustion. Many have limited memory and will run out of RAM for a NAT table well before hitting the 64k limit.

Greg Dubicki
  • 1,191
  • 1
  • 14
  • 30
user8162
  • 270
  • 2
  • 9
  • That's a very lengthy article. Can you summarize the parts which are relevant for this question? – Michael Hampton Sep 26 '13 at 01:23
  • First: The article is indeed very good (if lengthy). I can't find the specifc to my answer in my reading of it, but i suspect the answer may be hiding behind the mention of binding and mapping methods... and more specifically potentially the "endpoint independent" methods. Secondly, we have the luxury of not being limited by consumer grade hardware. We have pretty beefy Cisco Nexus 7000 series equipment at the edge. While I can't find any datasheets that give a MAX Connection limit... I assume it is well above the 65536 limit. – Daniel Widrick Sep 26 '13 at 01:47
  • 1
    http://stackoverflow.com/questions/2332741/what-is-the-theoretical-maximum-number-of-open-tcp-connections-that-a-modern-lin – hookenz Sep 26 '13 at 01:54
  • Sorry if I was unclear -- the answer depends on what kind of architecture the NAT box uses. If it's just a Linux host (or one of many commercial products based on the Linux IP stack), then yes, 64k connections per Internet-facing IP address is the theoretical limit. Again, though, there are memory constraints with NAT table sizes that may mean the practical limit is much smaller than 64k. – user8162 Sep 26 '13 at 02:31
  • Matt's link eventually bounced me over to: http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t Under the "Connect() Returning EADDRINUSE?" section, it would appear that the poster is implying that a socket can be bound to a given 'source ip x source port' and connect() can be called to various 'remote ip x remote port' combinations. This all relies on SO_REUSEADDR and SO_REUSEPORT options... but i am still not fully clear on if this means that the sockets must be LISTENING or if outbound connections can use SO_REUSEPORT. – Daniel Widrick Sep 26 '13 at 02:34
  • 1
    You asked about "other resource limits" being an issue with NAT. Besides my own experience of NAT boxes (both enterprise and soho grade) handling < 64k connections, there's also [Cisco documentation suggesting that memory and CPU constraints come into play.](http://www.cisco.com/en/US/tech/tk648/tk361/technologies_tech_note09186a0080094c32.shtml#na) – user8162 Sep 26 '13 at 02:53
  • @User8162 I suppose you are correct, I was hoping for a quantified 'x connections requires y memory, and z cpu cycles' but that is likely drifting too close to the load testing and capacity planning question. I would still like an answer to the SO_REUSEPORT portion I mentioned in the comment here, but maybe that is better suited for stack overflow ? – Daniel Widrick Sep 26 '13 at 03:03
2

5) Will our LAN be limited to 65536 (or some other theoretical limit) concurrent connections to the outside world through a single public IP address?

No, because one port NAT IP can be used for multiple connections:

cat /proc/net/ip_conntrack | grep 51380
tcp      6 191 ESTABLISHED src=10.1.8.5 dst=17.133.254.23 sport=51380 dport=5223 src=17.133.254.23 dst=my.nat.pub.ip sport=5223 dport=51380 [ASSURED] mark=0 use=2
tcp      6 24 CLOSE_WAIT src=10.1.26.1 dst=80.68.255.71 sport=51380 dport=80 src=80.68.255.71 dst=my.nat.pub.ip sport=80 dport=51380 [ASSURED] mark=0 use=2
Maximus
  • 21
  • 1
2

The long-story-short is that it's heavily platform-dependent, configuration-dependent, and implementation-dependent.

But let me explain quickly:

Apparently, other answers state about the theory limit reaching >65535 (notice port 0 is often reserved), which migth be true to certain extent such as...:

  • In big CGNAT systems or similar high-grade routers, whose main purpose is this one, including NAT-PAT.
  • In certain Linux distro might be possible in theory when using a PC for software NAT/routing by CPU under certains circumstances such as ram > 1GB, kernel prepared, etc.

However in the real world, where hardware-accelerated routing takes place with limited resouce, the NAT table has a well-known limit, which is often a configuration parameter for protection.

  • Cisco mentions starting at IOS 12, the max NAT depends on DRAM resulting on around ~10K translations (source), which is less than those 65K in your question.

  • Take your old xDSL router, if you want to bring up a P2P at home with many connections, most of them have configured a global max-limit of 1024~4096. For instance my high-end FTTH router at home has NAT limit configured to 8K by the vendor.

Finally, to answer Q2-rewrite, I have seen products with dispair implementations of RFC3489 with the following NAT tables. Obviously the last one does limit considerably the NAT posiblities:

  • iAddr:iPort - eAddr:ePort - dAddr:dPort (typical symmetric nat)
  • iAddr:port - eAddr:port - dAddr (very low end product)

Tumbs up if you like this answer!

Community
  • 1
JCM
  • 143
  • 1
  • 6
0

The primary key of the NAT mapping table: 

sourceIP:sourcePort + DestinationIP:DestinationPort

We can map different sourceIp:sourcePort to the same DestinationIP:DestinationPort. So port number(2^16) is not the limit for connection.

g10guang
  • 101
  • 1
  • I wonder how much memory per mapping? I guess 12 bytes for the map (32 + 16 + 32 + 16 bits), another 48 bits for the LAN machines mac address. Then there is ipv6. Hmmm. – Tomachi Apr 03 '20 at 17:41