AWS ECS: Service + autoscaling vs User Data launching Task

Question

Trying to wrap my head around ECS using EC2 launch (not Fargate, at least for now).

Assume a single long-running task that I want to launch with every container instance. The intended mechanism for this appears to be a Service, which will launch as many task instances as I configure - in this case, I start off with the same number of tasks as container instances.

Autoscaling seems to exist here in two layers: the cluster can autoscale its instances, and the Service can autoscale the number of Tasks. There does not seem to be any way to synchronize these two layers without Fargate.

One approach I came up with for getting around this limitation is to have scale-out triggered by a CloudWatch alarm on the cluster, and then use a lifecycle hook on the instance creation to trigger scale-out on the Service. I still haven't worked out how this would work for scale-in - is there some automatically generated event for Service scale-in?

then

I saw an AWS guide to starting a Task on container launch. The crux of the matter would seem to be this MIME User Data:

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
# Specify the cluster that the container instance should register into cluster=your_cluster_name

# Write the cluster configuration variable to the ecs.config file
# (add any other configuration variables here also)
echo ECS_CLUSTER=$cluster >> /etc/ecs/ecs.config

# Install the AWS CLI and the jq JSON parser
yum install -y aws-cli jq

--==BOUNDARY==
Content-Type: text/upstart-job; charset="us-ascii"

#upstart-job
description "Amazon EC2 Container Service (start task on instance boot)"
author "Amazon Web Services"
start on started ecs

script
    exec 2>>/var/log/ecs/ecs-start-task.log
    set -x
    until curl -s http://localhost:51678/v1/metadata
    do
        sleep 1
    done

    # Grab the container instance ARN and AWS region from instance metadata
    instance_arn=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | 
.ContainerInstanceArn' | awk -F/ '{print $NF}' )
    cluster=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .Cluster' | awk -F/ '{print $NF}' )
    region=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F: '{print $4}')

    # Specify the task definition to run at launch
    task_definition=my_task_def

    # Run the AWS CLI start-task command to start your task on this container instance
    aws ecs start-task --cluster $cluster --task-definition $task_definition --container-instances $instance_arn --started-by $instance_arn --region $region
end script
--==BOUNDARY==--

While there is a disclaimer at the beginning of the article, I understood it to mean that this method of using awscli to run a Task would mitigate all concerns.

Did I miss something? Is this a viable alternative to a Service with autoscaling?

score 0 · Accepted Answer · answered Mar 12 '18 at 08:29

After discussing this with a support guy from AWS, this seems like a pretty good solution for my issue.

In order to set this up, I needed to create an autoscaling group using a launch configuration that had as its basis an Amazon ECS-optimized AMI, and assign it a role that has ECS permissions. Then I was able to include the userdata as shown. When everything was configured properly (and the instances had public internet access), the instances would register themselves to my ECS cluster on launch, and they would run my task.

This would NOT be a good solution for having a container that lives independently of the task(s) you want to run, or for a task that can be restarted when it fails without restarting the whole container.

AWS ECS: Service + autoscaling vs User Data launching Task

1 Answers1