2

We are in the process of setting up aws cloudwatch exporter(https://github.com/prometheus/cloudwatch_exporter) to export cloudwatch metrics to our prometheus server. When reading through the documentation, I see a note about convergence of aws cloudwatch metrics:

"CloudWatch has been observed to sometimes take minutes for reported values to converge. The default delay_seconds will result in data that is at least 10 minutes old being requested to mitigate this."

However, I couldn't find any documentation on aws regarding what convergence of metrics is or how long it takes to happen. What exactly is convergence and how soon is it safe to scrape metrics of aws cloudwatch and into prometheus?

RichVel
  • 3,524
  • 1
  • 17
  • 23
johnzac92
  • 21
  • 1

1 Answers1

2

I have not found any official AWS documentation saying so, but this is true and happens for ELB/ALB at least, which I have observed.

The "convergence of metrics" means it may take up to 10 mins for AWS resources to report back to CloudWatch with the data.

For example, ALB may take up to 2 mins to report all the RequestCount. So, if you ask the CloudWatch API SUM on RequestCount for the last minute of one of the ALBs it may give you 500 (let's say). But if you re-ask, it will give you 1200 (let's say) as not all metrics were reported to CloudWatch instantly.

See this issue: https://github.com/ivx/yet-another-cloudwatch-exporter/issues/35

If you are using cloudwatch_exporter, keep an eye on API calls which the exporter makes as it can incur huge bills and AWS may block your account for sending too many API calls.

Yet Another CloudWatch Exporter solves both the above problems.

Nice Read: https://sysdig.com/blog/improving-prometheus-cloudwatch-exporter/

RichVel
  • 3,524
  • 1
  • 17
  • 23
Shubham Choudhary
  • 21
  • 3