How to add tag to Windows Server 2012 instances in AWS OpsWorks through Chef recipe?

Question

My employer has a need to add tags to AWS ec2 instances started in OpsWorks once they come online.

The instances are all MS Windows Server 2012 R2 Base in this stack. The goal is to add custom tags like "application":"foo" to the instances started in stack Foo and "application":"bar" in stack Bar.

I've found a github repo chef-aws-tag which I believe would do what I need if I ran Chef 11 on Linux. OpsWorks only does Windows Server 2012, and only with Chef 12.2.

When I add that chef-aws-tag recipe from the github repo to the Setup lifecycle event, the instance fails at the "running_setup" with "setup_failed".

The Chef failure log then reports this failure at the running_setup stage:

INFO: HTTP Request Returned 412 Precondition Failed : No such cookbook: aws 
ERROR: Running exception handlers
FATAL: Net::HTTPServerException: 412 "Precondition Failed "

This matches a dependency in metadata.rb, the line: depends 'aws', '>= 0.2.4'
I understand the dependency is missing. It's not clear to me how to fulfil this dependency. The recipe is in S3, Repository URL is https://s3-us-west-2.amazonaws.com/employer/ec2instance-tagging.zip

When I add the recipe to the Configure lifecycle event, the machine gets online without error but the tags are not added. The "stock" tags show up for the ec2 instance; the Keys "opsworks:instance" , "opsworks:layer:foo_layer" , "opsworks:stack" and "Name" all have Values I expect. The tags I want to add through the recipe ("application", "team", "environment") are not present.

The custom JSON is added at the Stack level:

{ "aws-tag": {
        "tags": {
            "team": "specialteam",
            "application": "foo",
            "environment": "development"
        }
    }
}

Is there a recipe or Cookbook specific to OpsWorks AWS I should be calling to set tags on instances after boot? How do I call that recipe?

I've read this AWS blog post on using OpsWorks to customize app deployment. Am I missing an identical "set aws tags on instance via chef" recipe or cookbook that is in the Amazon Web Services - Labs repository on GitHub?

The aws opsworks-cookbooks on github say "For Chef 12.2 Windows and Chef 12 Linux there are no built-in cookbooks" but I am hoping that is wrong :-) and that a cookbook has been added which I have not yet found. Alternatively, hopefully someone has done this already and documented it somewhere.

I am troubleshooting this following this AWS blog post Quickly Explore the Chef Environment in AWS OpsWorks (again written for Linux) and this SF question on debugging Chef on Opsworks

---

Answer 1

I was able to figure out some of this on my own. By opening a case with AWS Support I received enough pointers to resolve this.

Speed bumps to work around

1. Different cookbook directory structures for different platforms when using archive file.

   If you provide your recipes through an archive file stored on AWS S3, AWS requires one directory structure for Linux and a second, different, directory structure for Windows.

   Linux can put multiple cookbooks in one root directory. This is what $ berks package does by default.

   Windows does not accept that root directory. The $ berks package command does not have an option to build one archive with multiple cookbooks in the manner Windows accepts. I had to edit the tarfile manually.

2. Ruby on Windows

   There's an "error: Seahorse::Client::NetworkingError: SSL_connect" bug that may show up in the 'aws' cookbook. AWS wants https between services. This requires a working Certificate Authority bundle. Ruby on Windows cannot access the OS CA cert bundle. This can be fixed if:

   a) you install the aws-sdk gem, then

   b) set in the recipe the path to curl-ca-bundle.crt

3. Details on dependencies and community cookbooks.

   The documentation on AWS for Chef 12 on Windows does not mention Berksfile nor community cookbooks. These are documented on AWS for Chef 11 on Linux.

4. Specify region when calling update.

   Odd that running the recipe on the instance says the instance does not exist.

   FATAL: Aws::EC2::Errors::InvalidInstanceIDNotFound: aws_resource_tag[i-5ccd24d4] (stemsoft::default line 27) had an error: Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-5ccd24d4' does not exist

   AWS error codes says InvalidInstanceID.NotFound can be caused by a) being out of the region, or b) the ID of a recently created instance has not propagated through the system.

   Fix is to include region stack['region'] in the aws_resource_tag block of the recipe.

5. Correct policies attached to correct roles

   Ensure both 'aws-opsworks-ec2-role' and 'aws-opsworks-service-role' are set up with correct policy. 'aws-opsworks-service-role' can be left as-is, so OpsWorks Service can make API calls to other AWS services. 'aws-opsworks-ec2-role', the default IAM instance profile, needs permission on ec2 to create and describe tags.

Procedure

The goal of the steps below is to build an OpsWorks stack that runs Chef 12 on Microsoft Windows Server 2012, with a (custom) cookbook that contains a recipe to set a custom tag. The tag gets its info from custom json set in the stack in my example.

I have not found an existing cookbook that does this. I'd rather use an existing cookbook than build a custom cookbook.

At the end of this procedure, the recipe uses the chef community 'aws' cookbook to update a "team" tag from values set in custom json defined in the stack settings.

To do this, the recipe gets instance details using a search of existing data bag. It also gets region details using a search of a second existing data bag.

(It worked for me....)

1. At AWS, build stack
2. At AWS, assign custom json to stack
   • { "aws-tag": { "tags": { "team": "teamFooBar" } } }
3. At AWS, configure stack with custom repository. This example uses s3
4. At AWS, build layer
   • short name is all lower case
   • Security Group AWS-OpsWorks-Blank-Server
   • Recipes / Configure for 'Configure lifecycle event' For example, if your recipe is named onetime then the 'Configure lifecycle event' should contain the onetime::default recipe
5. Locally, install ChefDK (Chef Development Kit)
   • Locally, install Git for Windows
   • I like git-bash.exe
6. Locally, use Chef to initialize cookbook directory cookbook_dir
   • cd to build_root
   • within build_root make directory cookbook_dir
   • within build_root chef generate cookbook cookbook_dir
     • example: $ chef generate cookbook cookbook_dir --berks --copyright 'EMPLOYER' --email 'administrator@employerexample.com'
7. Locally, edit cookbook_dir\metadata.rb
   • add depends 'aws'
8. Locally, edit cookbook_dir\recipes\default.rb
   • '#{node['aws-tag']['tags']['team']}' will return "teamFooBar" in my example
   • edit recipe to call correct ca-bundle.crt
   • after edit, check recipe ruby syntax with $ ruby -c /path/to/cookbook_dir/recipes/default.rb
9. Locally, One Time Setup, initialize berksfile
   • in Git for Windows MinTTY cd to cookbook_dir and use $ berks install to initialize berksfile and track dependencies
10. Locally, edit cookbook_dir Berksfile
    • add cookbook 'aws'
11. Locally, cd into cookbook_dir and run $ berks package to build cookbook_dir.tar.gz
12. Locally, edit archive file for MS Windows Server 2012
    • unpack cookbook_dir.tar.gz . Move everything out of directory 'cookbooks' and up one level. Repackage cookbook_dir.tar.gz
13. upload cookbook_dir.tar.gz to s3
    • delete previous copy from s3
    • ensure permissions are correct ("Make Everything Public" as necessary)
14. At AWS, One Time Setup, edit role assigned to Instance started by OpsWorks the right to update tag
    • 'aws-opsworks-ec2-role' "Statement": [ { "Action": [ "ec2:CreateTags", "ec2:DescribeTags" ], "Resource": [ "*" ], "Effect": "Allow" } ]
15. Get instance going
    • add an instance
    • start an instance
    • stack > Run Command > Update Custom Cookbooks
16. Fix aws-sdk gem for Windows does not ship with proper ca-bundle.crt
    • install ruby
      • in initial set up, this step was needed
      • in a subsequent test run, this step was not needed
    • install aws-sdk
      • in initial set up, this step was needed
      • in a subsequent test run, this step was not needed
17. Confirm it works
    • check ec2 instance for presence of correct tag
    • check OpsWorks instance Logs (configure command)
    • Example of success: INFO: AWS: Updating the following tags for resource i-12345678 (skipping AWS tags): { .... "team"=>"teamFooBar"}

Example recipe

Example recipe example::default.rb

Chef::Log.info("********** from custom json, Team  is  '#{node['aws-tag']['tags']['team']}' **********")

instance = search("aws_opsworks_instance", "self:true").first
Chef::Log.info("Instance id from data bag: #{instance['ec2_instance_id']}" )

stack = search("aws_opsworks_stack").first
Chef::Log.info("**********  stack['region'] from data bag:  '#{stack['region']}' **********")

include_recipe 'aws'
chef_gem "aws-sdk" do
    compile_time false
    action :install
    version node[:aws][:aws_sdk_version]
end

ruby_block "Set the AWS Bundle" do
   block do
      require 'aws-sdk'
      Aws.config[:ssl_ca_bundle] = 'C:\ProgramData\Git\bin\curl-ca-bundle.crt'
   end
   action :run
end

aws_resource_tag instance['ec2_instance_id'] do
    tags('team' => node['aws-tag']['tags']['team']   )
    region stack['region']
    action :update
    Chef::Log.info("********** updated tags('team') **********")
end

---