Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?

Question

Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance.

We have two queues for two different tasks in Celery. One of these queues has tasks which will require an instance to be started in the autoscaling group. In the first autoscaling group which handles one kind of tasks, we always have one instance running. It is then easy to start another instance using Amazon CloudWatch. But for the other autoscaling group we don't want it to have any instances if there are no tasks because many a time there might not be tasks to be performed by this group. So only if the length of the Celery queue for this task goes above a given threshold, we want to start an instance in this autoscaling group. What would be a good way to do this?

I'd set up a cron to publish the queue length as a CloudWatch metric, and add auto-scaling rules to that metric. — ceejayoz, Jul 17 '15 at 14:23
Can we define our our CloiudWatch metric and publish to it? Where can I read more about this? — web_ninja, Jul 17 '15 at 14:27
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/publishingMetrics.html — ceejayoz, Jul 17 '15 at 14:28

score 8 · Accepted Answer · answered Jul 17 '15 at 14:34

8

I'd suggest a cron job to publish the current queue length as a CloudWatch custom metric, which will allow you to create auto-scaling alarms based on its value.

answered Jul 17 '15 at 14:34

ceejayoz

32,469
7
81
105

Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?

1 Answers1

Linked