Trouble bringing in CloudWatch data to Logstash

Question

Just getting started with a new ELK setup (never used it before, just trying to learn it). I have Logstash 2.2.4 running on ubuntu 14.04 LTS.

After putting a yaml file down with my monitor user's AWS credentials (policy configured as per the documentation via copy/paste), I created the following .conf file in /etc/logstash/conf.d:

input {
  cloudwatch {
   metrics => ["CPUUtilization"]
   filters => { "tag:Monitoring" => "Yes" }
   region => "us-east-1"
   namespace => "AWS/EC2"
   aws_credentials_file => "/var/opt/aws.yaml"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

I have three servers in us-east-1 with the tag "Monitoring" set to "Yes", but when I tail my logstash logs it is erroring out that there are no metrics to query. An example error entry (formatted for readability):

{
  :timestamp=>"2016-10-31T13:38:06.314000-0400", 
  :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  
  Plugin: <LogStash::Inputs::CloudWatch 
              metrics=>[\"CPUUtilization\"], 
              filters=>{\"tag:Monitoring\"=>\"Yes\"}, 
              region=>\"us-east-1\", 
              namespace=>\"AWS/EC2\",  
              aws_credentials_file=>\"/var/opt/aws.yaml\", 
              codec=><LogStash::Codecs::Plain charset=>\"UTF-8\">, 
                     use_ssl=>true, 
                     statistics=>[\"SampleCount\", \"Average\", \"Minimum\", \"Maximum\", \"Sum\"], 
                     interval=>900, 
                     period=>300, 
                     combined=>false>\n  Error: No metrics to query", :level=>:error}

EDIT

Based on the comment, I updated my policy for the service user whose credentials are in the above mentioned .yaml file. This did not change the behavior, here is what my policy currently looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444715676000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1444716576170",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        }
    ]
}

In checking that I had actually assigned the policy to the user correctly, I noticed that at some point in debugging I also assigned the 'CloudWatchReadOnlyAccess' policy, which looks like this (in case it breaks something):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "autoscaling:Describe*",
        "cloudwatch:Describe*",
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "logs:Get*",
        "logs:Describe*",
        "logs:TestMetricFilter",
        "sns:Get*",
        "sns:List*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

This is actually my first time using something other than the pre-defined policies, so I'm thinking perhaps I missed something in the writing of it.

Answer 1

What they don't tell you in the documentation, is that you need to have some more rights set in order to use filters. The code performs a describe-instances call on your account, using the filters to get a list of instance-ID's, which it then runs the cloudwatch API calls on.

The IAM policy listed in the documentation only covers the cloudwatch API calls. If you left off filters entirely in your config, you'd get data.

However, you want to filter on tags. For that, you'll need at least:

ec2:DescribeInstances
ec2:DescribeTags

In an allow statement to be able to get the data you need.

To troubleshoot this, I would verify that the AWS credentials do what they should to rule that out. The AWS CLI equivalent of what they're doing is:

aws ec2 describe-instances --filters "Name=tag:Monitoring,Value=Yes"

If that fails, then this is your problem. You might need ec2:DescribeNetworkInterfaces as well, but I'm not certain of that.

If that succeeds, then the problem is not with your EC2 rights, its with something else. You can replicate the call the plugin is making to cloudwatch this way:

aws cloudwatch list-metrics --namespace AWS/EC2 --dimensions "Name=InstanceId,Value=i-1234abcd" --metric-name CPUUtilization

The plugin is using the describe-instances call to fetch the InstanceId values for instances with the Monitoring tag.

If this works, and so does the instance-fetching, then it is a problem with the plugin somehow. Verify that the credentials file is actually readable by the logstash process. You can bypass the need to search for tags by attempting to fetch a specific instance. There is an example in the code for how to specify this.

filters => { 'instance-id' => 'i-1234abcd' }