0

My site receives numerous uploads and I can't afford to waste cycles and time processing them on the spot (e.g., trimming, optimizing, resizing, etc.). What technologies exist to delay and queue these non-essential commands for execution at a later time? Upon execution I would also like to use cpulimit to allocate a certain % and also limit memory usage.

Running Ubuntu 10.04 LTS

EEAA
  • 108,414
  • 18
  • 172
  • 242
MarkL
  • 13
  • 1
  • I edited your question to make it less like a shopping question and less subjective, both of which are off-topic. – EEAA Sep 27 '12 at 20:19

1 Answers1

0

Well, I'm not sure what the "best" is. You'll need to decide that for yourself.

You need some sort of message queue. There are many of these around: MSMQ, Gearman, RabbitMQ, Amazon SQS, etc.

Myself? I use RabbitMQ for communication between disparate parts of our application, and to queue up actions that should not be performed synchronously. It takes a bit of time to wrap your head around the difference between exchanges and queues, and to figure out how best to arrange things for your environment, but after that, it's a piece of cake to use.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Thanks. Reading about RabbitMQ now. I take it you run it in tandem with cron, which check to see if there are any messages queued and then executed them if there are? – MarkL Sep 27 '12 at 22:00
  • For delayed processing, yes you could do it that way, yes. Or, you have an RMQ consumer that is always bound and "listening" to the queue for messages. When a message is placed in the queue, it can act immediately. – EEAA Sep 27 '12 at 22:50
  • OK. You put me on the right path. I zeroed in on Redis + Resque, especially since I already use Redis as a memcache respacement. Thanks. – MarkL Sep 27 '12 at 23:31