6

I monitor the available disk space on Ubuntu servers using Nagios Core, NRPE and check_disk.

With previous Ubuntu versions, I used to get an output similar to this:

DISK OK - free space: / 43754 MB (80% inode=86%):

On Ubuntu 18.04.1, instead, I get:

DISK OK - free space: /var/tmp 43754 MB (80% inode=86%):

I am shown an incorrect mount point /var/tmp for the root / partition. I tracked this behaviour to be dependent on having PrivateTmp=true in nagios-nrpe-server.service:

  • I went to look into /var/tmp and found a directory named systemd-private-c5b5d3d362364af19af640147f2cb844-nagios-nrpe-server.service-4uILRy
  • then examined the service definition and noticed PrivateTmp=true (which isn't there e.g. for NRPE2 on Ubuntu 16.04)
  • finally, tried deleting the line and the root mountpoint was then detected as /

I feel like I'm facing three options:

  1. Live with it.

  2. Remove PrivateTmp=true.

  3. Find a reasonable workaround.

I'm inclined to just live with it, but if I were more aware of the implications of not having a private /tmp for the service, I could make an informed choice about option 2.

The optimal solution could be to find a workaround, instructing check_disk to return the correct mount point information even in this case. Not being able to access the system /tmp should not represent an obstacle.

Question: Please illustrate the implications of PrivateTmp=true, explaining why it would be recommended and in what cases and with what caveats it can be removed.

Secondary question: Is there a sensible workaround to make check_disk or an equivalent tool display the correct root mountpoint even when run by a service with PrivateTmp=true?


Additional info:

The complete command is: /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/vg-root. When run locally, even with the nagios user, the output correctly shows /. When run remotely from the Nagios server with: /usr/local/libexec/nagios/check_nrpe2 -H 192.168.1.2 -c check_root, the output shows /var/tmp instead of the expected /.

simlev
  • 1,037
  • 2
  • 14
  • 22

1 Answers1

3

I can reproduce this behavior by providing the block device to check_disk instead of the mount point.

For example:

root@cosmic:~# grep check_root /etc/nagios/nrpe.cfg 
command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda1
root@cosmic:~# /usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_root
DISK OK - free space: /var/tmp 6451 MB (68% inode=78%);| /var/tmp=3033MB;8010;9011;0;10013
root@cosmic:~# /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda1
DISK OK - free space: / 6451 MB (68% inode=78%);| /=3033MB;8010;9011;0;10013

But by using the mountpoint, I get the expected behavior:

root@cosmic:~# grep check_root /etc/nagios/nrpe.cfg 
command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
root@cosmic:~# /usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_root
DISK OK - free space: / 6451 MB (68% inode=78%);| /=3033MB;8010;9011;0;10013
root@cosmic:~# /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
DISK OK - free space: / 6451 MB (68% inode=78%);| /=3033MB;8010;9011;0;10013

This behavior does seem to be related somehow to setting PrivateTmp= in the systemd unit. When I remove this from nagios-nrpe-server.service, then check_disk returns the expected result when given the block device as well. I played around a bit with a trivial service that just ran /bin/df with PrivateTmp=true but I couldn't find any obvious problem there. It returned the correct results too.

I would suggest that the best thing to do, if you really need to be able to check disks by block device rather than mount point, is to report an issue to the Nagios NRPE developers so they can actually dig into the code and find whatever it is.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 1
    Great! I had thought about that, but now I don't see any reason why I couldn't use `-p /`. I believe I can drop a line to the Monitoring Plugins Development Team. – simlev Nov 15 '18 at 15:40