2

I'm wondered about how nginx handles tons active websocket connections? There is a lot of limitations, like a number of open files, maximum of 65k TCP connections between IP <-> (IP, port) and so on.

When I'm using nginx as reverse proxy and have, say, 5 nodes in a websocket's upstream, is there up to 65k active TCP connections between nginx and each node? In case if I have more than 300k active clients served by nginx, of course. Or nginx websocket proxy works in another way?

And my another question: which parameters (limits) should I tune to handle so many connections?

Ivan Velichko
  • 145
  • 2
  • 11
  • nginx has to have two open sockets for every webSocket that it is proxying, one connected to the client and one connected to the server. No other way around it. There are countless articles on the web about how to tune a particular OS for large numbers of open sockets. The advice is generally very specific to the specific OS. You can get way more than 65k open sockets if tuned appropriately. – jfriend00 Sep 17 '15 at 21:48

1 Answers1

4

Your understanding is correct as far as it goes, but your real limit is going to be file handles. Each socket connection requires a Linux file handle, and the default ulimit on those is 1024. /proc/sys/fs/file-max sets the limit for the entire system. You will need to raise these limits to handle high nginx connection rates.

The NGINX folks have tested up to 50,000 connections on a six-core server:

https://www.nginx.com/blog/nginx-websockets-performance/

The reality is that if you want tens of thousands of connections in the real world, you need to go to multiple reverse proxies behind a round-robin DNS. This is, for example, how Amazon's Elastic Load Balancer works. f you look for something on the AWS cloud like Slack.com and type 'nslookup slack.com', you'll get a long list of IP addresses. Type it again and you'll get a different list with the IP addresses rotated so a different one is at the head of the list. These are Amazon AWS ELB reverse proxies () on a round-robin DNS that forward requests to the actual application servers. The hard part at that point becomes managing registering and deregistering reverse proxies from the DNS as they come and go, or managing IP address takeover if that's what you intend to do. Hard problems, and thus why I use Amazon's solution rather than rolling my own.

eric.green
  • 385
  • 1
  • 4
  • Looks like the default file-max is now `26385181` on the latest Debian, which is nice. This seems a bit high though, I wonder why – NiCk Newman Jun 11 '16 at 18:33
  • file-max is the system limit. ulimit is the per-process limit. As you note, file-max will rarely be the problem on modern Linux, but you'll need to raise the per-process ulimit using your distribution's mechanism for doing so if you want more than 512 simultaneous connections with nginx. – eric.green Jan 24 '19 at 19:59