0

I've enabled the grok filter in our logstash configuration in order to parse logs from HAProxy using the %{HAPROXYHTTP} and %{HAPROXYTCP} patterns. This seems to work great and viewing the details for any log entry from haproxy I can see the various extracted fields (bytes_read, client_ip, client_port, termination_state, actconn, feconn, etc).

But from Kibana's "Create a new visualization" screen, none of these fields are available in the "Fields" popup when configuring the Y axis.

What do I need to do to make these extracted fields available for visualizations?

larsks
  • 41,276
  • 13
  • 117
  • 170

2 Answers2

1

Kibana independently tracks the ElasticSearch mappings so it can do translations between the ES datatypes and kinds of fields Kibana uses. That mapping list needs to be updated. You can find it in Management --> Index Patterns. Once you're in the Index Patterns list, select the index that you've added the fields to and click the roundy-roundy refresh button (next to the trash button, because of course). It'll remind you that it's resetting the popularity numbers for fields; but that's OK, you want it to see new ones.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
0

@sysadmin1138's answer was half the problem. The other half of the problem is that, for reasons that must have seemed good at the time but seem incredibly shortsighted now, the logstash patterns for haproxy don't provide explicit data types for fields. E.g., HAPROXYTCP is defined as:

HAPROXYTCP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}
\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termin
ation_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue}

Since, for example, bytes_read is defined as %{NOTSPACE:bytes_read}, it's a string data type and thus not available for visualizations. Fixing this means creating custom mappings in an index template before you populate late it with any data, so (a) toss all your existing data, and (b) figure out a list of all the fields you want to use that are mis-typed.

(NB: This also appears to be true for the httpd patterns, like %{HTTPD_COMMONLOG}. And probably for everything else as well.)

larsks
  • 41,276
  • 13
  • 117
  • 170
  • Oh, I've been bitten by that before. A trick I've used when I have a lot of a certain datatype that I need in visualizations/terms-queries is to update the ES template for logstash to set the fieldtype for those fields to `keyword` directly. – sysadmin1138 Apr 04 '20 at 20:13