AWS ECS service not starting any tasks

Question

I've set up a cluster, container, task and service on ECS.

After I created the service, it appears to be running with '3 desired tasks', however no tasks are being created as pending or running. Effectively it seems to be doing nothing.

My task definition (excluding nulls and empty arrays) is

{
  "containerDefinitions": [
    {
      "portMappings": [
        {
          "hostPort": 5000,
          "protocol": "tcp",
          "containerPort": 25565
        },
        {
          "hostPort": 5000,
          "protocol": "udp",
          "containerPort": 25565
        }
      ],
      "cpu": 0,
      "memoryReservation": 1024,
      "image": "itzg/bungeecord",
      "essential": true,
      "name": "BungeeCord"
    }
  ],
  "compatibilities": [
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-west-1:949960343466:task-definition/BungeeCordTask:3",
  "family": "BungeeCordTask",
  "requiresAttributes": [
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    }
  ],
  "requiresCompatibilities": [
    "EC2"
  ],
  "revision": 3,
  "status": "ACTIVE",
}

My cluster has 3 ECS instances, which are all running and have nothing on them: My cluster

My service is here: My service

I don't have any auto-scaling, load balancing or anything set up. But I do have some really default service discovery.

Anybody have any ideas why no tasks are starting?

Edit:

I've had a look at the "Stopped tasks" and it's actually empty.
I also SSH'd into each of my ec2 instances, and had a look through /var/log: I found a folder /var/log/ecs with some log files, but there are no errors or anything relating to my task specifically. It seems to be successfully polling a amazon web endpoint for updates.
I also ran curl google.com to verify that they have external access.

Just wondering if you found a solution to this problem, I've facing the same issue. — Sunny Agarwal, Feb 10 '20 at 09:27

score 3 · Answer 1 · answered Aug 11 '21 at 17:45

I'd check your Events tab in the ECS service. In some cases it will tell you there isn't enough memory to start the container. It's a bit tricky to find without knowing, since it's buried in a tab within the service. I'd expect some events must have occured.

score 1 · Answer 2 · answered Jan 05 '22 at 15:42

An answer that may help some. Similarly for me, the service was not attempting to start any tasks - no stopped tasks, no events, nothing showing up in CloudTrail.

It turned out that one of the subnets in the service definition no longer existed. The service silently failed to start any tasks with no evidence of the attempts being logged. So, if in doubt, try recreating the service to help ensure all the resources you're trying to use still exist.

score 1 · Answer 3 · edited Jun 25 '21 at 15:20

1

In the Clusters tab, pick the cluster. In the Service open the Tasks tab and switch the view to “Stopped tasks”.

You should see a list of tasks that failed to start and a reason why they failed.

Also check /var/log/amazon/{something about ecr}/* for other possible errors.

It could be that the instances can’t reach the docker registry (network config issue) or that they can’t login if it’s ECR (IAM Role) or needs username / password.

Hope that helps :)

edited Jun 25 '21 at 15:20

mikemaccana

3,070
5
24
29

answered Aug 05 '19 at 07:12

MLu

23,798
5
54
81

Hi, thanks for your response :) Please check my edits to the question. – Jack3311 Aug 05 '19 at 07:27

score 0 · Answer 4 · edited Oct 22 '19 at 15:03

0

It could be that the instances are not registered with ECS. Make sure to create instances only by using CloudFormation. Update the services only after all instances are in running state. If you are updating CloudFormation using the command line, make sure that you put in a sleep and allow your EC2 instances to startup first and then update the ECS service.

edited Oct 22 '19 at 15:03

kenlukas

2,886
2
14
25

answered Oct 22 '19 at 05:01

Sushrut Kanetkar

101

AWS ECS service not starting any tasks

4 Answers4