How to auto-scale ec2 instances with an encrypted root volume?

Question

I am trying to configure an autoscaling setup in AWS where the Node Launch Template includes encrypting the root volume (EBS). I have configured a service linked role, and a CMK in Amazon KMS with an IAM policy as per the documentation.

However, I am getting the following error when the ASG looks to create the instances:

Launching a new EC2 instance: i-0123456789xxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch

The troubleshooting documentation simply points at the original documentation and suggests the IAM policy isn't configured properly - I'm struggling to work out what's not right though.

The service linked role is configured on the ASG: SLR on ASG, and the SLR has the correct permissions in the IAM policy for the key that was used to encrypt the volume:

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: Encrypt",
            "kms: Decrypt",
            "kms: ReEncrypt*",
            "kms: GenerateDataKey*",
            "kms: DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: CreateGrant",
            "kms: ListGrants",
            "kms: RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

Note that launching the same AMI manually, specifying the root volume is encrypted with the same key, works. This suggests a problem with the SLR perhaps?

Or, do I need to create an AMI where the root volume is already encrypted?

UPDATE 11/05/2020:

Turns out there was a formatting error - there is a space after each colon in the Actions section. Removing that has fixed it and it works as expected now.

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

score 0 · Answer 1 · answered Mar 16 '20 at 15:25

0

I had the same problem and resolved it by adding the Service-Linked Role for Auto Scaling to the Key policy of the pertinent key (AWS Console -> KMS -> Customer managed keys -> YOUR_KEY -> 'edit' under the Key policy tab) as follows:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::READCTED:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
                    "arn:aws:iam::REDACTED:root"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

answered Mar 16 '20 at 15:25

Rafał Król

1
2

Sorry for the delay in responding. Turns out, the example I'd copied above, I modified to remove the space after the `:` character in the Action sections. Doing that has made this work as expected. Strange that I didn't get any sort of formatting error when applying it, but that's done the trick. – TommyKTheDJ May 11 '20 at 16:53
I would mark as an answer, but looking through, the privileges you've given the SLR are more than required. So yes, this fixes the error but at the cost of breaking the principle of least privilege. – TommyKTheDJ May 12 '20 at 07:18
No worries, good to hear you were able to figure it out. – Rafał Król May 15 '20 at 10:16

score 0 · Accepted Answer · answered May 11 '20 at 16:57

Turns out there was a formatting error - there is a space after each colon in the Actions section. Removing that has fixed it and it works as expected now.

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

How to auto-scale ec2 instances with an encrypted root volume?

2 Answers2