`pidof` check on host fails when docker container runs same app (e.g. apache2)

Question

The apache2 init script does a pidof check to detect if apache is already running.

    if pidof $DAEMON > /dev/null 2>&1 ; then
            if [ -e $PIDFILE ] && pidof $DAEMON | tr ' ' '\n' | grep -w $(cat $PIDFILE) > /dev/null 2>&1 ; then
                    AP_RET=2
            else
                    AP_RET=1
            fi
...
            elif [ $AP_RET = 1 ] ; then
                    APACHE2_INIT_MESSAGE="There are processes named 'apache2' running which do not match your pid file which are left untouched in the name of safety, Please review the situation by hand".

(File: /etc/init/apache2 on Ubuntu 16.04.3 LTS - truncated for brevity)

However, on a docker host, the VM containers may already have apache in them. In this case pidof returns non-empty even if there is no apache running on the host.

$ sudo service apache2 stop
$ pidof apache2
32742 32480 32379 32365 31295 31294 31293 31292 31291 31274 31270

This means that the init script succeeds only when all docker containers with apache in them have stopped (or not yet started). Consequently, apache on the host cannot be restarted.

How does one fix this situation so that the host's apache can be restarted independently of the VMs? Is there a version of pidof that will only detect pids owned directly by init?

A.B · Accepted Answer · 2017-11-09T19:54:17.457

Too bad there's a # can't use pidofproc from LSB here in the init script, without real explanation. I'd still consider this apache2 script to have a bug worthy of a report.

TL;DR: solution: replace pidof apache2 with pgrep --ns 1 ^apache2$ (or if this doesn't work, pgrep --ns 1 --nslist uts ^apache2$)

Long explanation about namespaces with an example that I wrote before finding pgrep could do it following:

Once you have the "candidates" using pidof, here's a method to separate them: check their namespaces, and compare them to pid 1 (init/systemd)'s namespaces. Example using lxc and the inetd process, but this is container's technology and process' name agnostic:

# lxc-start stretch-amd64
# pidof inetd
10285 3372
# ls -l /proc/1/ns/
total 0
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 cgroup -> cgroup:[4026531835]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 net -> net:[4026531993]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 pid -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 pid_for_children -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 nov.   9 19:49 uts -> uts:[4026531838]
# ls -l /proc/3372/ns/
total 0
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 cgroup -> cgroup:[4026531835]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 net -> net:[4026531993]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 pid -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 pid_for_children -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 nov.   9 19:51 uts -> uts:[4026531838]
# ls -l /proc/10285/ns/
total 0
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 cgroup -> cgroup:[4026532516]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 ipc -> ipc:[4026532415]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 mnt -> mnt:[4026532410]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 net -> net:[4026532418]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 pid -> pid:[4026532416]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 pid_for_children -> pid:[4026532416]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 nov.   9 19:50 uts -> uts:[4026532414]

Here it's clearly visible that pid 3372 shares pid 1's namespaces. 3372 is running on the host. 10285 doesn't share any namespace (ok user is the same: container run as root), so it's in a container. It's possible that sometimes some program running on the host has some of these changed for some reason (usually security related), but what shouldn't is the uts (hostname) namespace. So here's a script using stat and that given the process' name in arg "$1" (eg: set -- inetd or a script's argument) will give only the process in the same uts namespace, usually meaning the (same) host.

pid1uts="$(stat -c %N /proc/1/ns/uts|cut -d' ' -f3)"
for i in $(pidof "$1"); do
    if [ "$pid1uts" = "$(stat -c %N /proc/$i/ns/uts|cut -d' ' -f3)" ]; then
        echo $i
    fi
done | xargs -r

which in my example, returns 3372.

I explained how to do it, but why reinvent the wheel when pgrep has options to handle it:

# pgrep ^inetd$
3372
10285
# pgrep --ns 1 --nslist uts ^inetd$
3372

Or for most of the cases just:

# pgrep --ns 1 ^inetd$
3372

score 0 · Answer 2 · answered Nov 09 '17 at 06:27

0

if your service on host listening on port 80 you can figure out process id with netstats

#netstat -plan | grep :80

container processes should be bind with other port number on host and internally in container with 80 port. so you can easily figure out host process and kill it.

answered Nov 09 '17 at 06:27

Sunil Bhoi

189
1
1
9

As a SysAdmin, I know how to distinguish between processes in `netstat`, `lsof`, etc. This needs to be a scriptable solution. – tudor -Reinstate Monica- Nov 09 '17 at 22:26

`pidof` check on host fails when docker container runs same app (e.g. apache2)

2 Answers2