How to add AWS ELB access logs to logstash with S3 input?

Question

I'm using an ELK stack for server monitoring. My application's access logs which are from AWS ELB are stored in AWS S3. I am trying to add them to logstash with the following input:

input {
  s3 {
    access_key_id => "my_id"
    secret_access_key => "my_key"
    bucket => "my_bucket"
    region => "region"
    prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/**/**/**/*.log"
    type => "elb"
  }
}

But nothing is added to logstash, what am I doing wrong? I there another way to add them to logstash? The filter and output parts seems OK so I don't post them.

Note: I am using logstash 2.1 version

Does the IAM user you use have enough rights to get logs from S3? — Tom, Jan 20 '16 at 08:55
@apanagiotou Did you manage to get this to work? Why are you using type => elb and not "string"? — Eran Chetzroni, Apr 11 '16 at 06:57

score 0 · Answer 1 · edited Feb 08 '17 at 12:29

0

Could you add sincedb_path => "/tmp/alb-sincedb" and leave prefix like prefix => "AWSLogs/"?

Also it would be great if you install the latest version of input s3 gem. I believe I saw the changes there related to the iterating of objects inside of the bucket using the prefix by using V2 resources API. My current version is logstash-input-s3-3.1.2.gem.

edited Feb 08 '17 at 12:29

473183469

1,350
1
12
23

answered Feb 02 '17 at 08:27

oivoodoo

101
3

score -1 · Accepted Answer · answered Apr 11 '16 at 08:30

Seems like the prefix should be, as the plugin does not support globs (*) :

 prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/"

Full example:

input {
  s3 {
    access_key_id => "my_id"
    secret_access_key => "my_key"
    bucket => "my_bucket"
    region => "region"
    prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/"
  }
}

How to add AWS ELB access logs to logstash with S3 input?

2 Answers2