HaProxy suddenly crashing on new Ubuntu Images (same config works elsewhere)

Question

I have been using HAProxy for about two years with the same working configuration. I've installed HaProxy multiple times over the last week on Ubuntu 14.04, 15.10 and I am getting out of Memory crashes in HaProxy. It no longer works. I have attached the configuration I have been using as well as the errors I am seeing in syslog below.

I do have some ubuntu servers that I created a couple months ago that run the same version of HAProxy that do work with the exact configuration, but not sure the best way to pinpoint the issue.

Version information:

HA-Proxy version 1.5.14 2015/07/02
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

You can substitute www.mywebsite.com with anything such as www.cnn.com if you want to try it.

haproxy.cfg

global
    daemon
    maxconn 256000
    user haproxy
    group haproxy
    stats socket /var/run/haproxy.sock mode 0600 level admin
    pidfile /var/run/haproxy.pid
    spread-checks 5

defaults
    maxconn 256000
    log global
    option redispatch
    option allbackups
    option abortonclose
    option http-server-close
    timeout connect  5000
    timeout client   5000
    timeout server   5000
    timeout queue    5000
    timeout check    5000
    retries 2

# Statistics
listen stats :1936
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth myuser:mypass

# Front-end HTTP Rules
frontend f_http
    bind *:80
    mode http
    default_backend b_http_us

# Frontend HTTPS Rules
frontend f_https
    bind *:443
    mode tcp
    default_backend b_https_us

backend b_http_us
    mode http
    option accept-invalid-http-response
    balance roundrobin
    server www.mywebsite.com www.mywebsite.com:80 maxconn 100000 check inter 2s fastinter 2s downinter 2s

backend b_https_us
    mode tcp
    balance roundrobin
    server www.mywebsite.com www.mywebsite.com:443 maxconn 100000 check inter 2s fastinter 2s downinter 2s

Here is my syslog

Mar  8 10:42:05 www kernel: [  263.005325] gmain invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar  8 10:42:05 www kernel: [  263.005331] gmain cpuset=/ mems_allowed=0
Mar  8 10:42:05 www kernel: [  263.005340] CPU: 0 PID: 581 Comm: gmain Not tainted 4.2.0-27-generic #32-Ubuntu
Mar  8 10:42:05 www kernel: [  263.005341] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Mar  8 10:42:05 www kernel: [  263.005344]  0000000000000000 000000001c9253e8 ffff8800399bf958 ffffffff817eae99
Mar  8 10:42:05 www kernel: [  263.005347]  0000000000000000 ffff88003da1e040 ffff8800399bf9e8 ffffffff817e8c8f
Mar  8 10:42:05 www kernel: [  263.005349]  ffffffff810c43f1 0000000000000000 0000000000000001 0000000000000000
Mar  8 10:42:05 www kernel: [  263.005352] Call Trace:
Mar  8 10:42:05 www kernel: [  263.005364]  [<ffffffff817eae99>] dump_stack+0x45/0x57
Mar  8 10:42:05 www kernel: [  263.005367]  [<ffffffff817e8c8f>] dump_header+0x8c/0x214
Mar  8 10:42:05 www kernel: [  263.005374]  [<ffffffff810c43f1>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Mar  8 10:42:05 www kernel: [  263.005381]  [<ffffffff81185fa1>] oom_kill_process+0x1e1/0x3b0
Mar  8 10:42:05 www kernel: [  263.005383]  [<ffffffff81186726>] out_of_memory+0x556/0x5b0
Mar  8 10:42:05 www kernel: [  263.005399]  [<ffffffff8118c3c1>] __alloc_pages_nodemask+0x8c1/0xa10
Mar  8 10:42:05 www kernel: [  263.005406]  [<ffffffff811d37c1>] alloc_pages_current+0x91/0x100
Mar  8 10:42:05 www kernel: [  263.005409]  [<ffffffff81182510>] __page_cache_alloc+0xa0/0xc0
Mar  8 10:42:05 www kernel: [  263.005419]  [<ffffffff81211d90>] ? poll_select_copy_remaining+0x140/0x140
Mar  8 10:42:05 www kernel: [  263.005422]  [<ffffffff81184a4a>] filemap_fault+0x14a/0x3f0
Mar  8 10:42:05 www kernel: [  263.005425]  [<ffffffff81211d90>] ? poll_select_copy_remaining+0x140/0x140
Mar  8 10:42:05 www kernel: [  263.005430]  [<ffffffff811b0a80>] __do_fault+0x50/0xe0
Mar  8 10:42:05 www kernel: [  263.005433]  [<ffffffff811b5aab>] handle_mm_fault+0xf5b/0x17e0
Mar  8 10:42:05 www kernel: [  263.005440]  [<ffffffff81067a87>] __do_page_fault+0x197/0x400
Mar  8 10:42:05 www kernel: [  263.005442]  [<ffffffff81067d57>] trace_do_page_fault+0x37/0xa0
Mar  8 10:42:05 www kernel: [  263.005446]  [<ffffffff8105fe19>] do_async_page_fault+0x19/0x70
Mar  8 10:42:05 www kernel: [  263.005451]  [<ffffffff817f3bf8>] async_page_fault+0x28/0x30
Mar  8 10:42:05 www kernel: [  263.005453] Mem-Info:
Mar  8 10:42:05 www kernel: [  263.005457] active_anon:203766 inactive_anon:36 isolated_anon:0
Mar  8 10:42:05 www kernel: [  263.005457]  active_file:25 inactive_file:22 isolated_file:0
Mar  8 10:42:05 www kernel: [  263.005457]  unevictable:0 dirty:1 writeback:0 unstable:0
Mar  8 10:42:05 www kernel: [  263.005457]  slab_reclaimable:11917 slab_unreclaimable:32447
Mar  8 10:42:05 www kernel: [  263.005457]  mapped:31 shmem:138 pagetables:849 bounce:0
Mar  8 10:42:05 www kernel: [  263.005457]  free:1947 free_pcp:96 free_cma:0
Mar  8 10:42:05 www kernel: [  263.005461] Node 0 DMA free:3960kB min:60kB low:72kB high:88kB active_anon:10040kB inactive_anon:4kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:20kB slab_reclaimable:432kB slab_unreclaimable:1188kB kernel_stack:0kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? yes
Mar  8 10:42:05 www kernel: [  263.005466] lowmem_reserve[]: 0 975 975 975
Mar  8 10:42:05 www kernel: [  263.005468] Node 0 DMA32 free:3828kB min:3828kB low:4784kB high:5740kB active_anon:805024kB inactive_anon:140kB active_file:100kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032184kB managed:1000836kB mlocked:0kB dirty:4kB writeback:0kB mapped:120kB shmem:532kB slab_reclaimable:47236kB slab_unreclaimable:128600kB kernel_stack:1248kB pagetables:3300kB unstable:0kB bounce:0kB free_pcp:384kB local_pcp:384kB free_cma:0kB writeback_tmp:0kB pages_scanned:1116 all_unreclaimable? yes
Mar  8 10:42:05 www kernel: [  263.005473] lowmem_reserve[]: 0 0 0 0
Mar  8 10:42:05 www kernel: [  263.005475] Node 0 DMA: 2*4kB (EM) 11*8kB (UM) 4*16kB (UEM) 1*32kB (E) 1*64kB (U) 1*128kB (U) 0*256kB 1*512kB (E) 1*1024kB (E) 1*2048kB (M) 0*4096kB = 3968kB
Mar  8 10:42:05 www kernel: [  263.005485] Node 0 DMA32: 9*4kB (UE) 2*8kB (EM) 66*16kB (M) 85*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3828kB
Mar  8 10:42:05 www kernel: [  263.005493] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar  8 10:42:05 www kernel: [  263.005494] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar  8 10:42:05 www kernel: [  263.005495] 182 total pagecache pages
Mar  8 10:42:05 www kernel: [  263.005497] 0 pages in swap cache
Mar  8 10:42:05 www kernel: [  263.005499] Swap cache stats: add 0, delete 0, find 0/0
Mar  8 10:42:05 www kernel: [  263.005499] Free swap  = 0kB
Mar  8 10:42:05 www kernel: [  263.005500] Total swap = 0kB
Mar  8 10:42:05 www kernel: [  263.005501] 262044 pages RAM
Mar  8 10:42:05 www kernel: [  263.005502] 0 pages HighMem/MovableOnly
Mar  8 10:42:05 www kernel: [  263.005503] 7858 pages reserved
Mar  8 10:42:05 www kernel: [  263.005503] 0 pages cma reserved
Mar  8 10:42:05 www kernel: [  263.005504] 0 pages hwpoisoned
Mar  8 10:42:05 www kernel: [  263.005505] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar  8 10:42:05 www kernel: [  263.005510] [  204]     0   204     8826       85      23       3        0             0 systemd-journal
Mar  8 10:42:05 www kernel: [  263.005512] [  210]     0   210    10945      148      24       3        0         -1000 systemd-udevd
Mar  8 10:42:05 www kernel: [  263.005514] [  334]   100   334    25074       64      20       3        0             0 systemd-timesyn
Mar  8 10:42:05 www kernel: [  263.005516] [  542]     0   542     7286       56      20       3        0             0 cron
Mar  8 10:42:05 www kernel: [  263.005518] [  564]     0   564    70809      187      43       3        0             0 accounts-daemon
Mar  8 10:42:05 www kernel: [  263.005520] [  575]     0   575     7155       77      18       3        0             0 systemd-logind
Mar  8 10:42:05 www kernel: [  263.005521] [  577]   104   577    64093      161      28       3        0             0 rsyslogd
Mar  8 10:42:05 www kernel: [  263.005523] [  585]   105   585    10712      112      26       3        0          -900 dbus-daemon
Mar  8 10:42:05 www kernel: [  263.005534] [  658]     0   658    17482      187      37       3        0         -1000 sshd
Mar  8 10:42:05 www kernel: [  263.005537] [  662]     0   662     4024       37      13       3        0             0 agetty
Mar  8 10:42:05 www kernel: [  263.005539] [  713]     0   713     1056       19       7       3        0             0 haproxy-systemd
Mar  8 10:42:05 www kernel: [  263.005540] [  716]   108   716    15492     1223      26       3        0             0 haproxy
Mar  8 10:42:05 www kernel: [  263.005542] [  717]   108   717   212748   200030     411       3        0             0 haproxy
Mar  8 10:42:05 www kernel: [  263.005544] [  718]     0   718    24943      244      49       3        0             0 sshd
Mar  8 10:42:05 www kernel: [  263.005546] [  720]     0   720     9184      202      23       3        0             0 systemd
Mar  8 10:42:05 www kernel: [  263.005547] [  721]     0   721    14719      419      33       3        0             0 (sd-pam)
Mar  8 10:42:05 www kernel: [  263.005549] [  740]     0   740     5615      402      15       3        0             0 bash
Mar  8 10:42:05 www kernel: [  263.005551] [  761]     0   761     1876       19       9       3        0             0 tail
Mar  8 10:42:05 www kernel: [  263.005552] Out of memory: Kill process 717 (haproxy) score 788 or sacrifice child
Mar  8 10:42:05 www kernel: [  263.005612] Killed process 717 (haproxy) total-vm:850992kB, anon-rss:800120kB, file-rss:0kB
Mar  8 10:42:06 www systemd[1]: haproxy.service: Service hold-off time over, scheduling restart.
Mar  8 10:42:06 www systemd[1]: Stopped HAProxy Load Balancer.
Mar  8 10:42:06 www systemd[1]: Starting HAProxy Load Balancer...
Mar  8 10:42:06 www systemd[1]: Started HAProxy Load Balancer.

This isn't technically a crash... the process is being killed because the system has no memory available. The question appears to be why HAProxy appears to need so much. — Michael - sqlbot, Mar 08 '16 at 16:24
The process that triggered OOM was not the process that was killed. You need to set `vm.oom_kill_allocating_task` if you want the task which caused the OOM condition to die. — Michael Hampton, Mar 08 '16 at 16:47
if I remove the user and group from the config file, running as root, I am not seeing a crash, but a HTTP Error 504. I've tried proxying several sites and for some reason, it doesn't want to connect. — Kladskull, Mar 08 '16 at 17:24
Ok, so I should have looked at the hosts file... Digital ocean takes the server description/host name and adds a hosts file entry with 127.0.0.1 - messing my desire to proxy an external web server. — Kladskull, Mar 08 '16 at 17:33

score 0 · Answer 1 · answered Mar 09 '16 at 18:04

0

Ok, so I should have looked at the hosts file... Digital ocean takes the server description/host name and adds a hosts file entry with 127.0.0.1 - messing my desire to proxy an external web server.

answered Mar 09 '16 at 18:04

Kladskull

1,265
5
15
32

HaProxy suddenly crashing on new Ubuntu Images (same config works elsewhere)

1 Answers1