1

Environment, system details, and tools:

Has anyone ever seen this? We have a cron job that pushes Cloudwatch metrics from inside an instance by basically doing these steps:

  1. Get instanceId by running "wget -q -O - http://169.254.169.254/latest/meta-data/instance-id"
  2. Collecting some metric or other and building an AWS CLI query using aws cloudwatch put-metric-data ...
  3. Repeat

The weird thing we are seeing is, very infrequently, one of these runs will die after the wget query, with no output. As if the Metadata service just failed to respond.

Example end-of-script (we set bash -e and -x to die and to gather debug output):

++ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCE_ID=

The script ends there and exits because presumably wget exited with a non-zero exit status.

This is not reproducible, but it happens on the order of once every 2 weeks.

JDS
  • 2,508
  • 4
  • 29
  • 48
  • It seems like the first thing you'd want to do is to not suppress the actual error you're hitting, which you're doing by specifying `-q`. The output from `-O -` goes to `STDOUT` but all the other nonsense wget normally spews on a successful request goes to `STDERR`, so there isn't an actual need to suppress it when you're capturing its output. – Michael - sqlbot May 03 '16 at 21:10
  • ...although you could reduce the spew somewhat by trimming some known nonsense from the `STDERR` stream with something like `INSTANCE_ID=$(wget -O - http://169.254.169.254/latest/meta-data/instance-id 2> >(perl -pe 'undef $_ if /^Length/ || /^Saving\ to/ || /written\ to\ stdout/ || /100%/ || /^$/' >&2))` ... or just use `INSTANCE_ID=$(ec2metadata --instance-id)`, which might also provide a useful error -- not sure, since I've never encountered this problem. – Michael - sqlbot May 03 '16 at 21:17
  • Thanks. My colleague and I just discussed removing `-q`, so we'll probably do that. Beyond that though, I'm going to wrap this call in a retry loop and call it a day. Thanks – JDS May 03 '16 at 21:26

1 Answers1

3

You could be throttled. Specially if the issue is inconsistent. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-throttling

Mohammad Badrah
  • 131
  • 2
  • Welcome to ServerFault. https://meta.stackexchange.com/questions/8231/are-answers-that-just-contain-links-elsewhere-really-good-answers Can you make your answer stand on it own? – chicks Apr 29 '17 at 00:04