2

I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue size comes down. I have scripts to take care of introducing new boxes and making sure right configuration is running on them. I plan to check the queue by redis-cli after frequent(15 mins) intervals and if size is greater then threshold bring down the time gap between check(say 3 mins) check twice and then roll out new boxes and follow the same approach to kill the new servers.

I am new to celery and it would be great if I can get some advise on this approach.

Thanks

APZ
  • 954
  • 2
  • 12
  • 24

1 Answers1

2

If you are using an EC2 autoscaling group, this can be quite simple. I'm doing something similar, but with RabbitMQ queue length.

Check queue length periodically, and then decide for a certain queue length, how many workers you want. You can then use the ASG API (or command line tools) to set the "desired capacity" for the group according to the number of workers you need. Then the ASG takes care of creating and terminating instances as needed.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Thanks for the comment, no , am not using EC2 autoscaling group but I get the general approach you have opted for. – APZ Apr 12 '13 at 21:13
  • In that case, you *should* be using them. They're *very* nice. – EEAA Apr 12 '13 at 21:21
  • Sure I will definitely look in to EC2 ASG, one quick question before I start evaluation ASG, my code deployment depends on a tag value associated with each system and every time a new server of a particular type is launched we increase the tag value by 1, if using ASG we roll out new boxes can we specify a tag to be associated with the new box and tag's value be last_know+1. – APZ Apr 12 '13 at 22:00