11

I'm trying to download some files from S3 to an EC2 instance using the AWS CLI (1.8.7) on startup using a User-Data Script. This instance has a IAM Role & Instance Profile with the appropriate permissions. Sometimes, the CLI will fail with the error "Unable to locate credentials". It happens often, (not every time), but always after a separate S3 command, which worked. Here's the output of the script:

++ date +%r
03:24:10 AM
++ aws s3 cp s3://non-public-bucket-1/15mb-zip-file.zip ./15mb-zip-file.zip
Completed 1 of 1 part(s) with 1 file(s) remaining^Mdownload: s3://non-public-bucket-1/15mb-zip-file.zip to 15mb-zip-file.zip
++ date +%r
03:24:14 AM
++ unzip 15mb-zip-file.zip

# Snip

++ date +%r
03:26:01 AM
++ curl http://169.254.169.254/latest/meta-data/instance-id
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    10  100    10    0     0  13831      0 --:--:-- --:--:-- --:--:-- 10000
i-12345678++ date +%r
03:26:01 AM
++ aws s3 cp s3://non-public-bucket-2/my-small-text-file.json ./output.json
download failed: s3://non-public-bucket-2/my-small-text-file.json to output.json Unable to locate credentials

++ date +%r
03:26:34 AM

Any idea what could cause this?

Edit: This is an Amazon Linux server.

Simon Paris
  • 111
  • 1
  • 1
  • 5
  • Have you tried putting a 10 second delay before trying to fetch the files? Perhaps the IAM credentials aren't in place yet the first time the command is run. – EEAA Oct 23 '15 at 08:55
  • I'm having exactly the same issue with Ubuntu 16.04 LTS. – sj26 Aug 10 '16 at 11:24

1 Answers1

4

I had the same issue, even contacted support and they could not provide an answer. Exporting credentials to environmental variables does not work for aws s3 ... commands. You have to either use the aws configure util or create a ~/.aws/config file like this one:

[default]
aws_access_key_id=foo
aws_secret_access_key=bar
region=us-west-2

Please reference AWS documentation at http://docs.aws.amazon.com/cli/latest/topic/config-vars.html

Not only did I find it failing to execute at all with the environment exports on EC2 launch via the User Data but also post launch from the command line on a fully deployed running instance.

Steven K7FAQ
  • 277
  • 2
  • 3
  • 12
  • What is `~` in this case? I tried `echo $USER` and `echo $HOME` in my User Data script and both of these variables come out blank. – davidrmcharles Sep 22 '21 at 20:53