I have a CPU intensive Celery task and within one of the task it can be further parallelized using joblib. By default, starting a worker with celery, will create a pool with number of max concurrency equal to number of CPUs/cores (which is 36 in my case).
My question is, using this configuration, does it mean that each worker process will have only 1 core to use and will not benefit from the parallelization of joblib? or will it use all the cores when there is no other task in the queue of the worker.
For example:
@app.task # picked by celery worker process
def a_task():
algo = Algo(n_jobs=5) # further parallelization in the task
....
