CentOS: Unable to read output from nrpe/snmp agent on remote server, it works locally though, what am I doing wrong?

Question

I've installed NRPE and SNMP on some server as I'd like to monitor this server with Nagios. I have a few checks which are already running properly and I get the output as intended. Today, I've installed the check_ifstatus Nagios command in order to monitor the state of the network interfaces cards on one of my mysql servers. While running check_ifstatus locally on the server I get the relevant information correctly:

[root@dbr4 nagios]# /usr/lib64/nagios/plugins/check_ifstatus -H localhost
    OK: host 'localhost', interfaces up: 4, down: 0, dormant: 0, excluded: 0, unused: 0    |up=4,down=0,dormant=0,excluded=0,unused=0
[root@dbr4 nagios]#

but when I run the command from my Nagios server I get the following error:

[root@monitorvm ~]# /usr/lib64/nagios/plugins/check_ifstatus -H amadbr4
CRITICAL: No response from remote host 'dbr4' for 1.3.6.1.2.1.2.2.1.8  with snmp version 1
[root@monitorvm ~]#

Does it mean I have to enable snmp on the server I want to check? Is there a way to use NRPE in order to run this check? SELINUX is disabled on all machines.

Answer 1

Why are you using NRPE to check things via SNMP? You can check those directly from the Nagios box, assuming snmp is listening on an external interface (which you seem to want, judging by the netstat comments, despite looking for the wrong port).

You shouldn't be using -n unless you have compiled NRPE without SSL support.

(Also, you don't need to specify -p 5666, since that is the default. Same with -t 10.)

1. Make sure the IP of your Nagios box is in the allowed_hosts list.
2. See if you can talk to the NRPE daemon at all by running "check_nrpe -H dbr4" with no more args. You should get back the NRPE version number
3. Check that your "command[check_ifstatus]" line in nrpe.cfg matches your manual test.

Answer 2

Keith's answer is the most correct; you don't need to use NRPE to do SNMP checks, and it adds unnecessary complexity.

That said, the most likely immediate problem here is SELinux, unless you know you've turned it off. Run getenforce on the NRPE host; if it says "Enforcing", SELinux is enabled. If SELinux is enabled, run the check_nrpe command from your Nagios host and then check /var/log/audit/audit.log on the NRPE host. If there are lines in there with "type=AVC" that say "avc: denied", you're probably hitting an SELinux restriction.

The easy way to avoid this is to turn SELinux off by editing /etc/selinux/config and setting SELINUX to permissive (if you plan to reenable SELinux on this system at some point) or disabled (if you're sure you won't want to run SELinux on this system again). You obviously lose the benefits of SELinux if you do this, of course.

You might be able to address this by changing the SELinux context of the check_ifstatus plugin. You can check its context with ls -Z. On my system (with the standard targeted policy), it has context "system_u:object_r:nagios_system_plugin_exec_t:s0", of which "nagios_system_plugin_exec_t" is the important part. If setting it to "nagios_system_plugin_exec_t" (with chcon -t nagios_system_plugin_exec_t check_ifstatus) doesn't work, try with "nagios_unconfined_plugin_exec_t". If either of those changes works, you'll have to make them permanent with semanage fcontext -a -t nagios_system_plugin_exec_t '/usr/lib(64)?/nagios/plugins/check_ifstatus'.

If none of that works, you're probably running into some of the limitations on processes using the loopback address and you'll have to look at the AVC messages and craft your own SELinux policies. I haven't done enough of that to offer advice (aside from "check SNMP directly from your Nagios host; that's what SNMP is for"), but there are a handful of references online about doing this. YOu might want to look into audit2allow, which helps the process of making adaptations to an SELinux policy.