8

We are using RabbitMQ as message queue, and Celery for task queues, and I am trying to wrap my head around queues.

Could somebody possibly explain the operational differences between a task queue and a message queue to me?

So far, I get that the message queue will be used for actual data, like a text message or log snippet. It is placed on the message queue so a worker can collect it and do something with it, be that parsing, storing etc.

The task queue portion confuses me, not sure why or when or what that would do?

I would think workers (AKA tasks?) would constantly be polling the message queue for things to do, so why is there a task queue?

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Werner
  • 183
  • 1
  • 6

1 Answers1

6

Not quite, but close. The task queue in RabbitMQ is more akin to a task scheduler, like cron or Windows Task Scheduler.

The main idea behind Work Queues (aka: Task Queues) is to avoid doing a resource-intensive task immediately and having to wait for it to complete. Instead we schedule the task to be done later. We encapsulate a task as a message and send it to the queue. A worker process running in the background will pop the tasks and eventually execute the job. When you run many workers the tasks will be shared between them.

RabbitMQ is a message broker, hence the terminology. So operationally, functionally, there's really not much difference. One is for messages, the other is for tasks/jobs. The only real difference would be that messages are generally intended to processed (and therefore cleared out of the queue) as quickly as possible, whereas tasks are generally scheduled for a specific time, and therefore stay "queued" for a while. That's probably not a difference you'll have to worry about too much in the context of systems administration, though.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208