2

I've been dealing with an, arguably, strange issue on a AWS Elastic Beanstalk environment. I'm getting the following nginx error when there are more than 300 connections on a single instance at a given time:

2018/03/23 20:56:53 [error] 5431#0: *4121 connect() to unix:///var/run/puma/my_app.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: , server: _, request: "GET /api/v1/podcasts/ HTTP/1.1", upstream: "http://unix:///var/run/puma/my_app.sock:/api/v1/podcasts/", host:

My Puma config has

bind "unix:///var/run/puma/my_app.sock"

And the nginx config on the EC2 instance seems to have the socket properly set up (the default nginx config provided by EBS).

So yeah, I have no idea what's causing this. Any ideas? Thanks!

Icid
  • 73
  • 1
  • 7
  • 2
    You've probably hit a limit on open files. Raise the limit, or switch to TCP sockets. – Michael Hampton Mar 23 '18 at 21:30
  • @MichaelHampton Thanks! Could you give me an example on how to switch to TCP sockets? – Icid Mar 26 '18 at 10:25
  • 1
    Can you add outputs of (when it hits the 300 connections)`lsof | wc -l` and `ulimit -a`. If any Limits have been set in nginx share the config too. – Dextro67 Mar 29 '18 at 09:43
  • @Dextro67 Here's the entire setup -> https://gist.github.com/icidasset/931e80e72eee4c0d6aeafb6409b00cf8 I think the problem is that Beanstalk isn't picking up my custom nginx config and is still using the default one ... – Icid Mar 29 '18 at 14:05
  • @Dextro67 Ok, I managed to get the nginx config working, still the same error though... – Icid Mar 29 '18 at 16:28

1 Answers1

1

Well going through the issue and the configurations the failure is related to exceeded resource limit. Possibly two of the OS resources:

  • open file descriptors
  • processes available to a single user. Just to eliminate the process limitation keep your nproc value in EC2 instance to very high value or unlimited.

    sudo vi /etc/security/limits.conf

Refer this link for a brief idea. So since you cannot have worker_rlimit_nofile more than the ulimit set on the server first approch would be to calculate the nginx parameters and tweak as per requirement:

We can serve worker_processes * worker_connections / ( keepalive_timout * 2 ) users per second.

This scaling guide may help you quantifying the values you should increase and define. Once you are done defining the values for your web server you'll need to update the limits with higher or equal values. Check upto what user concurrency the server works to help you scale up.

Dextro67
  • 333
  • 2
  • 10