How do I debug an upstart job?

I have the following job in /etc/init/collector:

start on runlevel [2345]
stop on runlevel [!2345]

expect daemon

exec /usr/bin/twistd -y /path/to/my/tac/file

When I start the job with sudo service collector start, it hangs. If I ctrl-c and run initctl list, I see this:

collector start/killed, process 616

I can't see an instance of the twistd daemon in ps, and the HTTP server it's supposed to be providing does not exist.

I even tried this without 'expect daemon' and with a simple call to a one-line bash script using a script stanza, and it still doesn't work. I think I'm doing something very wrong. What could it be?

Cera

Posted 2012-03-23T02:50:26.657

Reputation: 327

No answer, but I'm struggling with a similar issue. It might even be a bug in Upstart: https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/438313 I don't know how to get from this hanging state other then rebooting. Also I believe it has to do with the expect daemon line.

– harm – 2012-04-02T16:10:20.150

Answers

You can redirect stdout and stderr of the entire shell using the script pragma (instead of exec) in conjunction with exec >FILE 2>&1, like so:

script
    exec >/path/to/some_log_file 2>&1
    exec your_command_here
end script

That should hopefully give you better insight into what's going on. I've found this useful for catching all sorts of problems in my upstart scripts. You could pipe your command's stdout/stderr directly, but you'll miss out on errors originating in the shell (like syntax errors).

On the other hand, if service is hanging, it might not even be hitting your script, in which case none of this will help, of course.

Charles

Posted 2012-03-23T02:50:26.657

Reputation: 286

Additionally, errors in the conf file itself are shown in dmesg. I found this out after I was not getting any output when using the above. I had made a typo in the chdir directive. – codingFoo – 2016-09-13T14:46:10.310

Where do you put that code? – kev – 2017-09-07T02:22:56.010

Thanks, this put me on the right track. In the end I piped the output to logger, so that I could just tail /var/log/syslog. – Cera – 2013-12-05T04:35:24.120

There's also the console log declarative, as defined here: http://upstart.ubuntu.com/cookbook/#console-log

I don't know enough about upstart to know if it's enabled by default, but you can enable it on a per upstart job basis, it'll by default output to /var/log/upstart/<job>.log

Ehtesh Choudhury

Posted 2012-03-23T02:50:26.657

Reputation: 1 330

Validate that upstart director exist , and add console log before the script phase. (in upstart version upper then 1.4 it's the default)

console log

script exec >/path/to/some_log_file 2>&1 exec your_command_here end script

For more info check thread: https://askubuntu.com/questions/207143/how-to-diagnose-upstart-errors/932155#932155

zachlaniado

Posted 2012-03-23T02:50:26.657

Reputation: 1