6

My team and I have recently experienced an odd issue regarding the AWS CLI. We are trying to use the CLI to start and stop several EC2 instances. While testing commands on single instances we have noticed that it is possible to 'stop-instances' using the CLI, but are unable to 'start-instances' using the CLI. We are able to start instances without any issue from the AWS console.

When attempting to start the instance from the CLI, the state switches from 'stopped' to 'pending', but shortly after it switches back to 'stopped'. The reason for that state is described as 'Client.InternalError: Client error on launch'. The access keys used have access to stop and start instance (running with dry-run option shows this).

Any ideas?

Amandil
  • 351
  • 1
  • 2
  • 7
  • So, to be clear, let's focus on a single example instance. You can start it and stop it repeatedly from the console, but not from the cli? It consistently fails to start from the cli and consistently succeeds at starting from the console? – Michael - sqlbot Nov 10 '16 at 13:23
  • That is correct - We can *stop* from CLI, but not start back. – Amandil Nov 10 '16 at 13:24
  • Note that it does not happen with all instances, just the ones we unfortunately happen to have to work with at the moment. – Amandil Nov 10 '16 at 13:25
  • 4
    Do these instances have encrypted EBS volumes? My rationale for asking is that if -- due to a non-obvious/hidden dependency -- the EBS volume's decryption key can't be obtained from KMS by EC2 (normally a transparent-to-you process), the instance could fail to start and there would apparently be no channel by which EBS could actually throw a meaningful exception. Are you using cli credentials from exactly the same IAM user that you're using on the console? Is the IAM user restricted in any policy to making requests only from certain IP addresses? – Michael - sqlbot Nov 10 '16 at 16:13
  • Yes, thank you! I've asked the administrator and that seems to be the issue. Now we just need to figure out what permissions to add to the role that executes the script. Would be great if you could help with that and post it as an answer. – Amandil Nov 10 '16 at 16:41
  • I can't post it until we figure out specifically what it is. But I'll see what I can find. – Michael - sqlbot Nov 10 '16 at 20:26

1 Answers1

6

As Michael has stated in the comments, the issue was permissions.

The only permission required was 'kms:CreateGrant' which has been added to the service user used to run the CLI commands.

Amandil
  • 351
  • 1
  • 2
  • 7