Logstash / Elasticsearch - trasform fields to lowercase in output conf

Question

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific.

As part of my FileBeats config, some standard fields are always generated as part of every message (and are location-specific), which I want to use as the basis for my my ES index:

output {
  elasticsearch {
    hosts => ["http://elasticsearch.mydomain.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}"
  }
}

However, ES is rejecting some of the indices as the field contains uppercase characters - identifier has acceptable values like myfoo but also could be MyBar:

[logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {"reason"=>"Invalid index name [logstash-2017.06.02-MyBar], must be lowercase"}

The casing isn't essential and I can add a mutate filter to forcibly lowercase the fields in question, but I would prefer to store the identifier field with proper casing, yet use the lower-cased version for the index name.

Is there function which can be called in the elasticsearch output, to lower-case the field in question? Something like

    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}.lowercase()"

score 4 · Answer 1 · answered Jun 24 '17 at 14:02

This can be done with a bit more mutate trickery.

Create a new field using mutate, set to your identifier.
In a second mutate, lowercase that new field.
Use the new field in your output.

Like so:

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[lc_identifier]}"
  }
}

You will end up with an lc_identifier field in your indices, but that shouldn't matter much.

score 0 · Answer 2 · answered Jul 29 '18 at 08:09

Building on sysadmin1138's answer:

You can use the @metadata field to prevent the lc_identifier field from appearing in your documents while indexed.

In your filter, change:

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

To:

filter {
  mutate {
    add_field => { "[@metadata][lc_identifier]" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "[@metadata][lc_identifier]" ]
  }
}

And to call it in your output:

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[@metadata][lc_identifier]}"
  }
}

This way you can create an index pattern with a lowercase identifier, but avoid having the lowercase field appear in the documents themselves.

score -1 · Answer 3 · edited Dec 25 '18 at 15:42

-1

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{lc_identifier}"
  }
}

edited Dec 25 '18 at 15:42

Thomas

4,155
5
21
28

answered Dec 25 '18 at 03:05

Long Đào Văn

1
1

Logstash / Elasticsearch - trasform fields to lowercase in output conf

3 Answers3