I use a combination of Logstash and the AWS Elasticsearch service to index S3 access logs.
The logs are collected in an S3 bucket, processed with the Logstash S3 input filter, renamed after they are processed and then archived in another bucket. I use this method so that the number of access log files that Logstash has to process in each rotation is as small as possible.
However, the logs are not being processed in real time. When I look at Kibana or query Elasticseach, the most recent log entry that I see will be the latest log entry from the previous hour. I never see log entries that are < 1 hour old.
I can't see anything in the s3 input configuration options to control this behaviour. There is an interval config option, which I have set to 120 secs. This is supposed to instruct Logstash to poll the S3 bucket which contains the logs every 2 mins.
I also use this Logstash system to process syslog input from a variety of servers, which does process logs in next to real time.
Is this something peculiar to the S3 input filter in Logstash?