17

I need to read from Rabbitmq and execute task in parallel using Celery in a single system.

[2014-12-30 15:54:22,374: INFO/Worker-1] ...   
[2014-12-30 15:54:23,401: INFO/Worker-1] ...
[2014-12-30 15:54:30,878: INFO/Worker-1] ...
[2014-12-30 15:54:32,209: INFO/Worker-1] ...
[2014-12-30 15:54:33,255: INFO/Worker-1] ...
[2014-12-30 15:54:48,445: INFO/Worker-1] ...
[2014-12-30 15:54:49,811: INFO/Worker-1] ...
[2014-12-30 15:54:50,903: INFO/Worker-1] ...
[2014-12-30 15:55:39,674: INFO/Worker-1] ...
[2014-12-30 15:55:41,024: INFO/Worker-1] ...
[2014-12-30 15:55:42,147: INFO/Worker-1] ...

It seams only 1 worker is running all the time .. ie one after another in sequential order. How can I configure Celery to run multiple workers to run parallel ?

SrC
  • 181
  • 1
  • 1
  • 4

3 Answers3

23

I have now updated my answer following the comment from MartinP regarding worker spawning child processes not threads:

Celery worker and worker processes are different things (Read this for reference).

When a worker is started it then spawns a certain number of child processes.

The default number of those processes is equal to a number of cores on that machine.

On Linux you can check the number of cores via:

$ nproc --all

Otherwise you can specify it yourself, for e.g.:

$ celery -A proj worker --loglevel=INFO --concurrency=2

In the above example there's one worker which will be able to spawn 2 child processes. It is normally advised to run a single worker per machine and the concurrency value will define how many processes will run in parallel, but if multiple workers required to run then you can start them like shown below:

$ celery -A proj worker -l info --concurrency=4 -n wkr1@hostname
$ celery -A proj worker -l info --concurrency=2 -n wkr2@hostname
$ celery -A proj worker -l info --concurrency=2 -n wkr3@hostname

Refer to celery docs for more info

andilabs
  • 105
  • 4
elnurcoot
  • 331
  • 2
  • 4
  • 3
    concurency parameter does't run threads. It run child processes by default so it process tasks in parallel - http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-c – MartinP Sep 28 '18 at 14:29
  • How do the concurrency and threads relate each other, if the default concurrency is the number of cores of the machine, what will be the number of threads? Is it configurable? – Thomas John Jan 04 '19 at 09:42
  • @ThomasJohn I _think_ the default number for threads is still the number of cores of the machine. Generally, you'll want to set the --concurrency flag high for thread-based workers. – FragLegs Sep 19 '19 at 18:22
2

Looks like your worker is just running a single process/thread. You probably just need to add the --concurrency or -c argument when starting the worker to spawn multiple (parallel) worker instances.

celery -A proj worker -c 4
Jamie B
  • 121
  • 3
0

I have found this

http://docs.celeryproject.org/en/latest/reference/celery.html?highlight=parallel

You can look for Canvas primitives there you can see how to make groups for parallel execution.

class celery.group(task1[, task2[, task3[, … taskN]]]) Creates a group of tasks to be executed in parallel.

Otherwise a good way is to go to the IRC channel and ask that special questions. Normally there are people who know that very good and they can help you.

René Höhle
  • 1,418
  • 3
  • 17
  • 26