HAProxy in front of logstash broken pipe

Question

I'm setting up ELK stacks with loadbalancing. Haproxy works fine for ES and Kibana but I'm having issues with Logstash.

Here's the haproxy configuration:

frontend logstash
   bind 0.0.0.0:5000
   mode tcp
   option tcpka
   option tcplog
   log global
   acl epa hdr_beg(host) -i epa-log
   acl tgops hdr_beg(host) -i tgops-log
   use_backend epa_log if epa
   use_backend tgops_log if tgops

backend epa_log
   mode tcp
   server elk01 elk01.example.org:5001 check
   server elk02 elk02.example.org:5001 check
   server elk03 elk03.example.org:5001 check

backend tgops_log
   mode tcp
   server elk01 elk01.example.org:5002 check
   server elk02 elk02.example.org:5002 check
   server elk03 elk03.example.org:5002 check

As you can see, im trying to have 1 frontend serving port 5000 and redirecting to a different port based on the url of the source.

If I do a netcat directly to the server it works perfectly

nc elk01 5001 -vv < /var/log/httpd/error_log

But if I use the haproxy frontend, it fails with a broken pipe error.

nc tgops-log 5000 -vv < /var/log/httpd/error_log
Ncat: Version 6.40 ( http://nmap.org/ncat )
libnsock nsi_new2(): nsi_new (IOD #1)
libnsock nsock_connect_tcp(): TCP connection requested to 10.129.10.2:5000 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [10.129.10.2:5000]
Ncat: Connected to 10.129.10.2:5000.
libnsock nsi_new2(): nsi_new (IOD #2)
libnsock nsock_read(): Read request from IOD #1 [10.129.10.2:5000] (timeout: -1ms) EID 18
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 26 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 35 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 42
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 42 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 51 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 58
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 58 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 67 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 74
libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [10.129.10.2:5000]
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 74 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 83 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 90
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 90 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 99 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 106
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 106 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 115 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 122
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 122 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 131 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 138
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 138 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE ERROR [Broken pipe (32)] for EID 147 [10.129.10.2:5000]
Ncat: Broken pipe.

If I check the TCP logs, I find that it somehow won't reach the backend, throws a NOSRV

 logstash logstash/<NOSRV> -1/-1/0 0 SC 15/0/0/0/0 0/0n 10.129.10.82:50724

I've tried different HAProxy options, like clitcpka, balance source, using only 1 backend and more but none resolves this.

score 1 · Accepted Answer · answered May 31 '17 at 15:56

1

You can't use mode tcp and match headers. mode tcp means don't treat the connection as HTTP and just pass the raw bytes through.

answered May 31 '17 at 15:56

longneck

22,793
4
50
84

So here i'd have no choice but to make a diffrent "listen" for every port? – Gab May 31 '17 at 16:01
Dispatching connections to different backends depending on the hostname which the client connected to is impossible to do on the TCP layer. - https://serverfault.com/questions/643131/proxying-tcp-by-hostname – Gab May 31 '17 at 17:59

HAProxy in front of logstash broken pipe

1 Answers1