HAProxy being killed with more that 54,000 connections

Question

I am trying to run HAProxy (1.4.8) on a EC2 machine running Ubuntu 10.04. I need HAProxy to be able to handle many thousands of long-running persistent connections (websockets). With the current setup HAProxy gets killed at around 54,300 connections (roughly).

If I am running HAProxy in the foreground, the only output is "Killed". Am I right in thinking this is the Kernel killing the process? Is this because it is out of resources? Can I increase the resources?

The CPU and memory consumption are low with 50,000 connections, so I don't suspect either of these.

How can I prevent this from happening?

check /proc/sys/fs/file-max. on different systems can be anywhere from 50,000 to 500,000 — Javier, Jan 12 '11 at 14:38
I upped the limit to 350000 like this `echo "350000" >> /proc/sys/fs/file-max` and the problem still persisted. — Olly, Jan 12 '11 at 15:03
Did you check the system logs `/var/log/messages` and `/var/log/syslog`? — Khaled, Jan 12 '11 at 15:35
This is the output from `/var/log/messages`: https://gist.github.com/776339 `Out of socket memory` seems like it might be relevant. — Olly, Jan 12 '11 at 15:56

score 2 · Answer 1 · answered Jan 13 '11 at 11:44

2

Switching to a 64-bit instance solved the problem. I'm guessing we were running into the per-process memory limit on a 32-bit instance.

answered Jan 13 '11 at 11:44

Olly

449
1
4
11

2

At 54000 concurrent connections, you should be careful about your TCP settings. If running with default settings (87kB read buffer, 16kB write buffer), you can end up eating 10 gigs of memory just for the sockets. It is very likely that in 32-bit, the system encountered memory allocation issues and decided to kill the most hungry user-space process, which "dmesg" should tell you. – Willy Tarreau Jan 19 '11 at 05:30
@WillyTarreau - could you point us to any information about how TCP settings should be adjusted for very high concurrency situations? (i.e. how to avoid such high memory usage). Thanks. – UpTheCreek Jan 14 '13 at 13:10
@WillyTarreau - ah sorry, I see there is some info in your post which ochoto linked to (http://serverfault.com/questions/106537/tuning-linux-ip-routing-parameters-secret-interval-and-tcp-mem/106555#106555) – UpTheCreek Jan 14 '13 at 13:13

score 2 · Answer 2 · edited Apr 13 '17 at 12:14

2

You should check this answer from Wily Tarreau to the very same question:

edited Apr 13 '17 at 12:14

Community

1

answered Jan 14 '11 at 12:56

Ochoto

1,174
7
12

HAProxy being killed with more that 54,000 connections

2 Answers2