2

So I have a shell script that daemonizes celery, and creates a bunch of workers running as a daemon. I would like to have a way to restart the celery task when the underlying source gets changed since the --autoreload option does not work.

According to my reading of the celery docs, kill -HUP $pid will kill the daemonized celery process and then make a new one with the same arguments. However, when I try it, celery goes down and doesn't come back up. Is something wrong with my command? Could celery be failing silently in the background when it starts (and if this is the case, where would I go to figure out what's wrong and see log output)?

Literal command is kill -HUP \`cat /var/run/celery/w1.pid\`. Checking ps aux | grep celery returns nothing. There is no logfile output at all after I send the kill signal.

Any ideas?

Kevin Meyer
  • 145
  • 7
  • Did you find any interesting messages in the log after shutting it down? It's good to mention that you've looked, even if you didn't find anything. – Andrew B Jan 22 '13 at 00:24
  • Nothing at all. Though daemonized celery has never been the greatest about failing loudly. Had a bunch of fun times where the log directory was owned by root, I was running celeryd as me, and celeryd was crashing (silently, after saying that it was running) every time it tried to write to the log. – Kevin Meyer Jan 22 '13 at 01:18

2 Answers2

3

The HUP handler in Celery simply starts the shutdown procedure and calls execv with the same arguments as the process started with when it completes. This is a pretty naive way, since it cannot know if it will come back up, but we haven't found a better solution that works in a signal handler yet.

If you use celery multi then it's better to use the celery multi restart command, which will also wait for the old worker to stop first (the generic-init.d script uses this for its restart command).

Here's the SIGHUP handler code: https://github.com/celery/celery/blob/master/celery/apps/worker.py#L280-L295 As you may see this requires sys.executable and sys.argv to be set so that it can be used to restart the worker (not sure why this isn't documented)

I don't know who started this trend, but people have come to expect SIGHUP to either reread configuration files or restart itself, but it's not always easy to do correctly and I think it was irresponsible to choose a signal that's sent when the terminal window closes ;) We have discussed removing the Celery HUP handler on several occasions.

asksol
  • 166
  • 3
  • Ok. So back to trying to edit celery by hand so I can load up workers from python instead of going back out to shell it is. Thanks. – Kevin Meyer Jan 22 '13 at 17:47
1

Are you sure that the documentation you're consulting is for the actual version deployed? If the HUP functionality was added in a newer version than the one you're using, the default behavior would probably be to terminate the process.

(I apologize if this is unhelpful, I have no experience with this particular daemon and it's the best I can offer.)

Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • celery is version 3.0.13. Docs are for version 3.0, and I believe that I saw the same note in the version 2.5 docs. Good thought though – Kevin Meyer Jan 22 '13 at 01:13