1

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch.

This feature has been successfully collected by MongoDB Input Plugin in real time.

However, the ElasticSearch Input Plugin does not start with the data that you recently collected when you run the collection. Duplicate data is created when collecting all data.

I would like to collect data from my most recent collection. How do I set it up?

There is another question.

The Mongo input plugin distinguishes recently collected data by "placeholder_db_dir", "placeholder_db_name" settings.

What is the ElasticSearch input plugin?

I am config file info.

input {
  elasticsearch {
    hosts => '192.168.10.101:9200'
    index => 'sample'
    scroll => '5m'
    docinfo => true
  }
}

filter {
        json {
                source => 'message'
        }
}

output {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "es"
        }
        stdout { codec => rubydebug }
        file {
                path => '/home/jskang/jskang-platform/logs/logstash/logstash-%{+YYYY.MM.dd}.log'
                codec => rubydebug
        }
}
junsung kang
  • 15
  • 1
  • 6

1 Answers1

2

If your intent is to capture changes to logstash in the last N minutes, you will need to manage your query and scheduling more directly. The elasticsearch input does not have the concept of a since_db, so you will need to do it through use of a structured query => and use a schedule =>. Such as...

query    => [your ES query, returning everything in the last 2 minutes]
schedule => "/2 * * * *"

This will run the input collection every 2 minutes, and return everything with a timestamp in the last 2 minutes.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296