AWS CloudFormation - Calling CLI from Init can't get credentials

Question

I'm using a CloudFormation template to boot up a custom windows AMI, which needs to retrieve some code from an S3 bucket and execute it, once it boots up.

After quickly being able to do it manually using the AWS CLI (s3 sync) I've been struggling to get this to work via CloudFormation for over 8 hours now, to no avail. In essence, this command always fails with:

"NoCredentialsError: Unable to locate credentials"

Initially I tried setting the /.aws/credentials and /.aws/config files, in many different ways, until I finally realized that the user identity running cfn-init.exe and it's child processes doesn't have access to these files at all and that won't work. So instead I've opted for setting environment variables using setx, but that doesn't seem to work either and I'm still getting the same error.

I'm getting quite frustrated with this as every test requires a change to the CF template, uploading to S3, creating a stack, waiting a good 5-10 minutes for it to finish bootstrapping only to RDP and find out it failed, again.

Here's the init > config portion of my template:

"commands" : {
  "0-Tester" : {
    "command" : "echo \"I am OK.\" > \"d:\\test.txt\""
  },
  "1-SetAK" : {
    "command" : { "Fn::Join" : ["", [
      "setx AWS_ACCESS_KEY_ID ",
      { "Ref": "AutomationUserAK" }
    ]] }
  },
  "2-SetSK" : {
    "command" : { "Fn::Join" : ["", [
      "setx AWS_SECRET_ACCESS_KEY ",
      { "Ref": "AutomationUserSK" }
    ]] }
  },
  "3-Pullcode" : {
    "command" : "aws s3 sync s3://some-s3-bucket d:/dev/ --debug"
  }
}

The first 3 commands (Tester, SetAK, SetSK) work just fine. The last one (Pullcode) fails, every time.

So at this point I'm assuming I'm going at it wrong. Maybe I need to configure a specific IAM for the CF stack, maybe I should use setx ... /M, maybe a million other options, but since this trial and error has been going for the entire length of my work day and then some I figured it won't hurt to ask.

Answer 1

Each of your commands is run in a separate PowerShell, hence setx AWS_ACCESS_KEY_ID sets the variable in one shell instance and exits. Then setx AWS_SECRET_ACCESS_KEY sets another variable in a new shell and exits. Then you run aws s3 sync in yet another shell where none of the previous variables exist and it fails because the credentials variables do not exist in that shell.

Instead of calling it from Metadata you can do something like:

"UserData": {
  "Fn::Base64": { "Fn::Join": ["", [
    "<script>\n",
    "cfn-init.exe -v -s [...]\n",
    "setx AWS_ACCESS_KEY_ID ", { "Ref": "AutomationUserAK" }, "\n",
    "setx AWS_SECRET_ACCESS_KEY ", { "Ref": "AutomationUserSK" }, "\n",
    "aws s3 sync s3://some-s3-bucket d:/dev/\n",
    "</script>"
  ] ] }
}

In this case it will all be run in the same shell instance and the keys will be still set when you call aws s3 sync.

Use IAM Roles instead

However I have to say that passing access and secret keys to CloudFormation template is a bad bad practice. Instead the CloudFormation template should create IAM Role with the appropriate permissions to access the S3 bucket and assign it to the EC2 instance. The aws tool on the instance will than have transparent access to S3 without any need to supply access and secret keys. Something like this should do for your CloudFormation template:

First create the IAM Role with S3 access policy (adjust the policy for your needs):

"InstanceRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "Service": [ "ec2.amazonaws.com" ] },
          "Action": [ "sts:AssumeRole" ]
        }
      ]
    },
    "Path": "/",
    "Policies": [
      { 
        "PolicyName": "S3_Access",
        "PolicyDocument": {
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [ "s3:List*", "s3:Get*" ],
              "Resource": "*"
            }
          ]
        }
      }
    ]
  }
},

Then create an IAM Instance Profile that refers to the above IAM Role...

"InstanceProfile": {
  "Type": "AWS::IAM::InstanceProfile",
  "Properties": {
    "Path": "/",
    "Roles": [ { "Ref": "InstanceRole" }
    ]
  }
},

And finally create the EC2 Instance that gets that IAM Profile assigned (and in turn IAM Role with the S3 access policy).

"Instance": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "IamInstanceProfile": { "Ref": "InstanceProfile" },
    [...],
    "UserData": {
      "Fn::Base64": { "Fn::Join": ["", [
        "<script>\n",
        "cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
        "</script>"
      ] ] }
    }
  },
  "Metadata": {
    "AWS::CloudFormation::Init": {
      "config": {
        "commands" : {
          "Pullcode" : {
            "command" : "aws s3 sync s3://some-s3-bucket d:/dev/ --debug"
          }
        }
      } 
    }
  }
},

When using IAM Roles the AWS credentials are dynamically generated and transparently obtained by the aws command. This is much more secure and a preferable way to give EC2 instances access to AWS resources.

Learn more about EC2 Instance Roles for example here: https://aws.nz/best-practice/ec2-instance-roles/

Hope that helps :)