0

So I have a fresh logstash install and I am trying to deploy logstash to get a handle on the logs.

I am going through and will eventually segment the logstash filters based on subsystem and currently I am working on parsing osd logs.

Here is a sample line I am working with:

2015-06-02 16:45:49.515277 7f4968cfe700  0 -- 10.16.64.68:6813/97613 >> 10.16.64.29:6805/35260 pipe(0x25e36500 sd=538 :6813 s=2 pgs=15426 cs=623 l=0 c=0x1586fa20).fault with nothing to send, going to standby

My filter currently looks like this:

%{TIMESTAMP_ISO8601:date} %{BASE16FLOAT:osd_epoch}  %{NUMBER:error_bool} -- %{CEPH_HOST:client_A} %{FROMTO} %{CEPH_HOST:client_B}

Where ${CEPH_HOST} and ${FROMTO} are just short patterns ::

FROMTO (?:[<|>]){1,2}
CEPH_HOST (%{IPORHOST:ip}\:%{POSINT:port}/%{POSINT:socket})

The issue is that IP now seems to house two addresses. 

  "client_A": [
[
  "10.16.64.68:6813/97613"
]
],
"ip": [
[
  "10.16.64.68",
  "10.16.64.29"
]
],
"HOSTNAME": [
[
  "10.16.64.68",
  "10.16.64.29"

I would like to grok it so that client_a has an IP and client_b has an IP.

However, does it matter in the end?

Can I leave it as is?

If so, will I be able to sort it later? If not, how do I segment it so that client_a and client_b are separated?

Do I need to create a "unique" pattern for both?

GregL
  • 9,030
  • 2
  • 24
  • 35
Lookcrabs
  • 21
  • 6
  • Based on that JSON output, your grok filter isn't working 100% since it seems to be missing some fields. That, or you've clipped it for brevity. If it's the latter, can you please post the whole thing? It will make things easier. – GregL Jun 03 '15 at 00:16
  • It was clipped for brevity. I am still making it but for now l just use %{GREEDYDATA} on the end. I am rather new to log stash, may I ask why this would be easier or if there is any other data I can provide to help. Thanks GregL! – Lookcrabs Jun 03 '15 at 21:49
  • I ask for more details because there might be things affecting the filter and output than are initially evident. The whole grok filter, as well as the whole resulting event would be great. – GregL Jun 03 '15 at 22:47

1 Answers1

0

I'm not sure if you ever worked out how to do this, but I was looking at something for my own environment that's similar and I think the only way to do it is to setup patterns for each client.

I've tested them both in the grok constructor and they return valid fields.

If you just wanted to have a field for Client A (10.16.64.68:6813/97613) and Client B (10.16.64.29:6805/35260), this would work:

%{TIMESTAMP_ISO8601:date} %{BASE16FLOAT:osd_epoch}  %{NUMBER:error_bool} -- (?<client_a>(%{IPORHOST}\:%{POSINT}/%{POSINT})) %{FROMTO} (?<client_b>(%{IPORHOST}\:%{POSINT}/%{POSINT}))

2015-06-02 16:45:49.515277 7f4968cfe700 0 -- 10.16.64.68:6813/97613 10.16.64.29:6805/35260 MATCHED
client_a: 10.16.64.68:6813/97613
client_b: 10.16.64.29:6805/35260
error_bool: 0
date: 2015-06-02·16:45:49.515277
osd_epoch: 7f4968cfe700

If you also wanted individual fields for each element of that connection (client a/b IP, client a/b port, client a/b socket) this pattern should work:

%{TIMESTAMP_ISO8601:date} %{BASE16FLOAT:osd_epoch}  %{NUMBER:error_bool} -- (?<client_a>(%{IPORHOST:client_a_ip}\:%{POSINT:client_a_port}/%{POSINT:client_a_socket})) %{FROMTO} (?<client_b>(%{IPORHOST:client_b_ip}\:%{POSINT:client_b_port}/%{POSINT:client_b_socket}))

2015-06-02 16:45:49.515277 7f4968cfe700 0 -- 10.16.64.68:6813/97613 10.16.64.29:6805/35260 MATCHED
client_a: 10.16.64.68:6813/97613
client_b: 10.16.64.29:6805/35260
client_a_ip: 10.16.64.68
client_b_ip: 10.16.64.29
client_a_port: 6813
client_b_port: 6805
client_a_socket: 97613
client_b_socket: 35260
error_bool: 0
date: 2015-06-02·16:45:49.515277
osd_epoch: 7f4968cfe700

GregL
  • 9,030
  • 2
  • 24
  • 35