Kibana @timestamp mapping & filter

Question

I'm using following system/package:

$ cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
$ rpm -q filebeat
filebeat-1.3.0-1.x86_64
$

with /etc/filebeat/filebeat.yml:

$ cat /etc/filebeat/filebeat.yml 
filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
      input_type: log
  registry_file: /var/lib/filebeat/registry
output:
  elasticsearch:
    hosts: ["localhost:9200"]
shipper:
logging:
  to_syslog: true
  files:
$

"message": "Sep 8 10:20:01 X CROND[11586]: (root) CMD (/usr/lib64/sa/sa1 1 1)",

How can I use timestamp from message as @timestamp in Kibana?
How can I separate out rest of message (daemon, etc.) into separate fields?

Please bear with me, if I didn't ask my question properly, I'm only learning) — alexus, Sep 08 '16 at 18:14

score 0 · Answer 1 · answered Sep 09 '16 at 03:47

The message field is text, not something Kibana knows how to use as a timestamp. You need to add some additional parsing in order to convert the timestamp from your log file into a date data type. You can learn more about Elasticsearch data types by reading the relevant documentation.

You've configured Filebeat to output directly to Elasticsearch. In order to parse the timestamp (and potentially other fields) from your log file, you'll need to configure Filebeat to output to Logstash instead. Logstash can then be used to modify your log data, parsing the timestamp as well as doing other pattern matching.

Having said all that, it looks like you might be using Filebeat to read a syslog file. That's fine, but it's also possible for Logstash to directly receive syslog data, which could simplify your overall setup.

I think the OP's question could be generalized as follows: Let's say he had a time field defined above and it contained some time stamp value. He wants Kibana to use that field as the official "timestamp" rather than defaulting to Elasticsearch's automatically-generated timestamp field. — Howiecamp, Jul 16 '17 at 02:38

score 0 · Answer 2 · answered Jan 09 '20 at 16:48

0

Case 1: If you have time stamp in the message as @timestamp then change it to root level by changing json.keys_under_root: true then change "json.overwrite_keys: true

Case 2: Add @timestamp to your app for example if your log has data as {"@timestamp":"2017-01-18T11:41:28.753Z","message":"Some log"}

Then change "json.overwrite_keys: false" to true in filebeat.yml and now the @timestamp matches

answered Jan 09 '20 at 16:48

Sahit

101
1

Hi, do you have any reference to this from the documentation? – Sidney Dec 03 '21 at 12:17
@Sidney you can refer the example here `https://discuss.elastic.co/t/in-filebeat-what-is-the-difference-between-json-keys-under-root-and-decode-json-fields/202732/2` For documentation, you can refer this page `https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html` – Sahit Dec 22 '21 at 06:01

Kibana @timestamp mapping & filter

2 Answers2