How to stop all child processes spawned by a service

Question

I have a service running on Ubuntu with the following configuration:

#/etc/init/my_service.conf

start on (local-filesystems and net-device-up IFACE=eth1)

respawn

exec python -u /opt/XYZ/my_prog.py 2>&1 \
                 | logger -t my_prog.py

On stopping the service with sudo service my_service stop, the python process is not killed. Killing the parent process with kill also does not kill the python process.

How do I completely stop the service (i.e all child processes killed)? Ideally, I would like to not modify the configuration file above.

Answer 1

Ideally, I would like to not modify the configuration file above.

Tough! It's the right thing to do.

You need to change your exec into script, and stop running that python program in a forked subprocess as part of a pipeline. This ServerFault answer explains how to do this in an embedded shell script. I'd make just one change to the script given there, in the last line:

exec python -u /opt/XYZ/my_prog.py 2>&1

There's no really good reason not to log standard error too, after all.

Ever more complex gyrations to cope with forking, from expect daemon to switching to systemd, miss the point that the right thing to do is to stop the daemon from forking. If there's one good thing to come out of the current kerfuffle, it's the continued confirmation that what IBM wrote and recommended in 1995 has been right all these years.

Get used to the idea of chain loading daemons. There are plenty of toolsets that make such things simple. Get used to the idea of not using shell scripts, too. There are plenty of toolsets that are designed specifically for this work, that eliminate the overheads of shells (which is a known good idea in the Ubuntu world).

For example: The shell commands in the ServerFault answer can be replaced with a script that uses Laurent Bercot's execline tools which are designed to be able to do this very thing without subshells and unlinked FIFOs:

#!/command/execlineb -PW
pipeline -w {
    logger -t my_prog.py
} 
fdmove -c 2 1 
python -u /opt/XYZ/my_prog.py

which you would then simply

exec /foo/this_execlineb_script

With my nosh toolset, it would similarly be a script containing:

#!/usr/local/bin/nosh
pipe 
fdmove -c 2 1 
python -u /opt/XYZ/my_prog.py | logger -t my_prog.py

Or alternatively, one could have this stanza directly in the Upstart job definition (using a trick to avoid shell metacharacters so that Upstart doesn't spawn a shell):

exec /usr/local/bin/exec pipe --separator SPLIT fdmove -c 2 1 python -u /opt/XYZ/my_prog.py SPLIT logger -t my_prog.py

Further reading

• Russ Allbery (2013-12-18). Bug#727708: upstart proposed policy in Debian. debian-ctte.
• Jonathan de Boyne Pollard (2001). "Don't fork() in order to 'put the dæmon into the background'.". Mistakes to avoid when designing Unix dæmon programs. Frequently Given Answers.

Answer 2

On GNU/Linux, ordinarily, there is no way to stop a service and all child processes it spawned, because child processes can change their PPID (parent process ID). The only way to know would be to trace the system calls to spawn processes as they are made, and keep a list of these processes.

The Ubuntu init system, upstart, does not do this. So the answer to your question is it's impossible -- on Ubuntu -- without:

1. Modifying that script;
2. Knowing exactly which process IDs are spawned by that process;
3. Keeping track of those process IDs manually;
4. Killing them individually.

This is why you should run a distribution of Linux that runs systemd. As you can see, systemd keeps track of all child processes and can kill every one of them with a single command. This is how GNU/Linux system administration should be, but because systemd is so new, and because it's "Not Invented Here" (meaning, Canonical didn't invent it), Ubuntu doesn't want to use it.

Answer 3

You have skipped the expect stanza. The upstart cookbook specifies:

Warning

This stanza is extremely important: read this section carefully!

Upstart will keep track of the process ID that it thinks belongs to a job. If a job has specified the instance stanza, Upstart will track the PIDs for each unique instance of that job.

If you do not specify the expect stanza, Upstart will track the life cycle of the first PID that it executes in the exec or script stanzas. However, most Unix services will "daemonize", meaning that they will create a new process (using fork(2)) which is a child of the initial process. Often services will "double fork" to ensure they have no association whatsoever with the initial process. (Note that no services will fork more than twice initially since there is no additional benefit in doing so).

In this case, Upstart must have a way to track it, so you can use expect fork, or expect daemon which allows Upstart to use ptrace(2) to "count forks".

To allow Upstart to determine the final process ID for a job, it needs to know how many times that process will call fork(2). Upstart itself cannot know the answer to this question since once a daemon is running, it could then fork a number of "worker" processes which could themselves fork any number of times. Upstart cannot be expected to know which PID is the "master" in this case, considering it does not know if worker processes will be created at all, let alone how many times, or how many times the process will fork initially. As such, it is necessary to tell Upstart which PID is the "master" or parent PID. This is achieved using the expect stanza.

You need this because the pipe | you are using is creating children processes. You can find in the book Advanced Linux programming a brief intro to that, where it is stated that:

For example, this shell command causes the shell to produce two child processes, one for ls and one for less:

 $ ls | less

What I do not know is whether this implies one or two forks, so that I would experiment with modifying the line respawn in your code with either

 expect fork
 respawn

or

 expect daemon
 respawn

I do not believe this can be achieved only with systemd, even though my logo clearly shows I am a fan of systemd.