My cluster is on AWS and I've used kops to build by cluster.
I'm trying to send the kubernetes logs to AWS Cloud-watch using fluentd. I was able to stand-up the fluentd pods.
I am seeing the below error from the pod's logs. I have created necessary policies for the pods to assume to send logs to the log-group in cloud watch logs.
2018-10-23 18:16:03 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-23 18:15:59 +0000 error_class="Aws::Errors::MissingCredentialsError" error="unable to sign
request without credentials set" plugin_id="object:2b07171c9ce4"
018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:72:in `sign_request'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:112:in `apply_signature'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:65:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/retry_errors.rb:138:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/json/handler.rb:11:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/request.rb:70:in `send_request'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-cloudwatchlogs-1.4.0/lib/aws-sdk-cloudwatchlogs/client.rb:736:in `describe_log_groups'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:403:in `log_group_exists?'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:152:in `block in write'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `each'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `write'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:354:in `write_chunk'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:333:in `pop'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/output.rb:342:in `try_flush'
2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/output.rb:149:in `run'
2018-10-23 18:16:07 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-23 18:16:01 +0000 error_class="Aws::Errors::MissingCredentialsError" error="unable to sign
request without credentials set" plugin_id="object:2b07171c9ce4"
2018-10-23 18:16:07 +0000 [warn]: suppressed same stacktrace
