27

I have this problem with NRPE, all the stuff I've found so far on the net seems to point me at things I've already tried.

# /usr/local/nagios/plugins/check_nrpe -H nrpeclient

gives

NRPE v2.12

as expected.

Running the command by hand (as defined in nrpe.cfg on "nrpeclient", gives the expected response

nrpe.cfg:

command[check_openmanage]=/usr/lib/nagios/plugins/additional/check_openmanage -s -e   -b ctrl_driver=0 bat_charge

"Expected response"

But if I try to run the command from the Nagios server I get the following:

# /usr/local/nagios/plugins/check_nrpe -H comxps -c check_openmanage
NRPE: Unable to read output

Can anyone think of anywhere else I might have made a mistake with this? I've done the same thing on multiple other servers with no problem. The only difference I can think of with this is that this box is RHEL 5 based, whereas the others are RHEL 4 based.

Those two bits above that I've tested are the what most people seem to suggest when people have had this problem.

I should mention that I get a weird error in the logs when I restart nrpe:

nrpe[14534]: Unable to open config file '/usr/local/nagios/etc/nrpe.cfg' for reading 
nrpe[14534]: Continuing with errors...
nrpe[14535]: Starting up daemon
nrpe[14535]: Warning: Daemon is configured to accept command arguments from clients!
nrpe[14535]: Listening for connections on port 5666 
nrpe[14535]: Allowing connections from: bodbck,combck,nam-bck

Even though, it's plainly reading that /usr/local/nagios/etc/nrpe.cfg file to get the stuff it's talking about further down..

Bart De Vos
  • 17,761
  • 6
  • 62
  • 81
ticktockhouse
  • 731
  • 1
  • 10
  • 17

20 Answers20

36

You have a rights problem.

Change the command to:

command[check_openmanage]=sudo /usr/lib/nagios/plugins/additional/check_openmanage -s -e -b ctrl_driver=0 bat_charge

(add sudo)

Then, add the nagios-user to the sudoers:

nagios ALL=(ALL) NOPASSWD:/usr/lib/nagios/plugins/additional/check_openmanage

Or you could just chmod the file... That also works.

If you are using CentOS, Red Hat, Scientific or Fedora, make sure to disable Defaults requiretty in the sudoers file.

Bart De Vos
  • 17,761
  • 6
  • 62
  • 81
  • 1
    @Bart De Vos but the answer you added will make a security hole > adding something to the sudoers file open you up for a potential security risk. i.e. if via a buffer overflow someone is able to drop a file with the same name in the same location, they could execute it without knowing the root password and gain control of the box :S Is there no way to somehow put a signature (SHA1 or MD5) of the application in the sudoers file to prevent that type of attack. i.e. the injected file won't have the same signature and thus doesn't execute. [Read the first comment here](http://crashatau.blogspot.co – Ahmad Hajjar Mar 16 '12 at 09:45
  • 1
    @X-Ware: While this is true, the chance a buffer overflow could be abused here is very slim. To prevent it from happening however, you should use apparmor/SELinux. That is why these things exist. – Bart De Vos Mar 16 '12 at 13:15
  • I guess different distros even have different users, in my case the user to add to visudo was npre not nagios. I still followed Bart De Vos's solution, however you can see which user is trying to access the nrpe command by viewing the /var/log/secure access log. Jul 24 15:39:09 hostname sudo: nrpe : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/vg_uxp-lv_root –  Jul 24 '14 at 19:43
  • @AhmadHajjar Are you serious? You think someone will hack nagios (a system thats 20 years old) and use that user to execute a file with root permissions. AND you think I didn't make the executable to be run as root as read-only to prevent someone from copying a file over it? If you are worried about it, instead of using sudo you can setuid the check_openmanage executable itself and let anyone run it! – Evan Langlois Jan 19 '17 at 12:05
11

Short answer : if you're using a Bash plugin, make sure you have a shebang stating which interpreter should be used : #!/bin/bash

I was facing the same issue with a Nagios plugin I wrote myself. The script was running as expected when launched locally, even when running as user nagios using following statement :

$ sudo sudo -s -u nagios
$ /path/to/my/plugin.sh
STATUS: OK

But remote-launching using NRPE from the Nagios3 server was unsuccessful :

$ /usr/lib/nagios/plugins/check_nrpe -H my-nagios-client -c my_plugin
NRPE: Unable to read output

I finally solved this case by adding a shebang in my script, as it appeared that running the script through NRPE did not used the same interpreter as when running through sudo sudo -s -u nagios.

Mickaël Le Baillif
  • 514
  • 5
  • 13
  • Had this problem when using a ruby script nagios plugin with rbenv. Fix was to create a wrapper script with `#!/bin/bash -el` `eval "$(rbenv init -)"` `/usr/lib/nagios/plugins/check_something $@` – TrinitronX Oct 03 '13 at 21:45
  • 1
    amazing answer! sudo -s -u nagios allowed me to see why nagios couldn't return output from a specific plugin. thank you so much! – ufk Jan 21 '15 at 15:00
6

In my case problem was simply - user nagios was not able to execute the script. After chmod it started to worked. Sudo is not necessary. Its even evil :)

bluszcz
  • 409
  • 1
  • 6
  • 16
  • 1
    The real answer is this. Nagios was not able to execute the script, either due to bad permissions, the script being mispelled, or the script not being there. – droope Oct 08 '13 at 02:46
5

check_nrpe was getting 'NRPE: Unable to read output' despite the check working locally because the plugin I was using did not work well with SELinux. Disable it and make sure to remove the file's contexts:

$ ls -l check_om_storage
-r-xr-xr--. 1 root nrpe 3808 Feb 27 17:54 check_om_chassis
$ setfattr -x security.selinux check_om_storage
$ ls -l check_om_chassis 
-r-xr-xr-- 1 root nrpe 3808 Feb 27 17:54 check_om_chassis
AXE Labs
  • 1,519
  • 5
  • 19
  • 24
5

In my case, only one plugin failed while several others worked okay. It turned out to be a LOCALE problem.

The plugin was check_mem.sh and it performed a grep for Mem in the output of free. But system wide LOCALE returned Speicher (German) instead of Mem, so all received values were empty strings.

MadHatter
  • 78,442
  • 20
  • 178
  • 229
Rush
  • 51
  • 1
  • 1
  • 2
    Rush, welcome to SF! This is an excellent first answer, in my opinion: short, to the point, and it adds something new to the collection of answers already here. +1 from me, and I hope to read more such answers from you in the future (I hope you will forgive my small formatting edit). – MadHatter Feb 24 '16 at 12:00
4

Check pathing, permissions, selinux, iptables.

Mine was a pathing issue in client:nrpe.cfg, double check the command path to the check_* plugin name. These can be confusing, (lib/local) (libexec/plugins) as a pathname. I mistakenly yanked and put the paths from the commented prepackaged nrpe cfg file to make commands. The make install or yum plugin install puts these in a difft directory.

commaneted: /usr/local/nagios/libexec/check_disk

versus

realpath: /usr/lib/nagios/plugins/check_disk

From the server I was able to confirm it was not a firewall issue, could telnet to the 5666 port, could run a blanket check_nrpe and get the status as a return value. Could run the commands locally but nrpe had the wrong path on the client in the nrpe.cfg.

hello world
  • 41
  • 1
2

This is a permission issue, just give the script execution right and it will be ok :

here an example : Before / Remote host:

[root@puppet1 nrpe.d]# ls -l /usr/lib/nagios/plugins/check_mem.sh
-rwxr--r-- 1 root root 1598 Jul  7 10:55 /usr/lib/nagios/plugins/check_mem.sh

NRPE server : 

[root plugins]# ./check_nrpe -H 172.19.9.200 -c check_mem_vb
NRPE: Unable to read output

After : Remote Host :

[root@puppet1 plugins]# chmod o+x /usr/lib/nagios/plugins/check_mem.sh

[root plugins]# ./check_nrpe -H 172.19.9.200 -c check_mem_vb
Memory: OK Total: 1980 MB - Used: 139 MB - 6% used|Total=2076479488;;;Used=145076224;;;Cache=1528111104;;Buffer=211890176;;;

Problem Resolved.

chicks
  • 3,639
  • 10
  • 26
  • 36
Youssef ASEBRIY
  • 21
  • 2
  • 1
    Good answer, but it's also good to note that allowing all users to run check_nrpe, as chmod o+x does, could be a potential security risk, depending on how the system is configured/accessed/used. – austinian Jul 08 '15 at 18:51
2

In case of custom NRPE plugins, make sure to print some output along with the exit value. If there is no output from the script NRPE will complain saying NRPE unable to read output. You can enable debugging in nrpe.cfg and observe this error.

If you can see correct output when running command directly by nagios user, but get unable to read output when running via NRPE, make sure the output you expect is not going to stderr. For example, ipmitool sel elist command will return SEL has no entries to stderr, and stderr is not captured by nrpe. You can add 2>&1 to redirect stderr to stdout in such case.

Gerald Schneider
  • 19,757
  • 8
  • 52
  • 79
Karthik
  • 114
  • 4
1

In my case the log file being monitored was owned by root:adm, so adding the nagios user to the adm group made the check_log command succeed, but only when executed directly on the monitored hosts. It continued to fail using check_nrpe on the Nagios server until I restarted the nagios-nrpe-server service on the monitored hosts, e.g.

service nagios-nrpe-server restart

So apparently restarting the service was necessary to make the permissions change take effect for NRPE, but it took me a while to figure this out.

Tony
  • 111
  • 5
1

In my case the issues was selinux related (running RHEL 6.5, selinux is set to enforcing).

Installing nagios-plugins-* via yum will create your plugin files in /usr/lib64/nagios/plugins. If you check the fcontext on those plugin files (ls -lZ), you will see the files have the context type set to "nagios_system_plugin_exec_t", which is the context type that check_nrpe expects.

In my case, I had created a custom script "check_mem.sh" using "vi". The resulting file had the context type set to "lib_t". This was causing nrpe to output the "NRPE: Unable to read output".

Changing the file context to "nagios_system_plugin_exec_t" solved the issue:

chcon -t nagios_system_plugin_exec_t /usr/lib64/nagios/plugins/check_mem.sh

The usual selinux troubleshooting would have pointed me to this issue as well (checking /var/log/audit/audit.log), but it was ofcourse the last thing I thought about

Edit: chcon just temporarily changes the context. To change it persistently use semanage fcontext -a -t nagios_system_plugin_exec_t /usr/lib64/nagios/plugins/check_mem.sh restorecon -vF /usr/lib64/nagios/plugins/check_mem.sh

Alexandru Todicescu
  • 11
  • 2
0

It could be that you haven't installed your Nagios plugins, NRPE cannot find them, or access them.

I have never had to add my commands to Sudoers. Make sure commands are owned by the Nagios user and they are readble.

Daniel Baker
  • 179
  • 2
0

I Think you must add the plugins in your local dir /usr/lib64/nagios/plugins/*. I had the same problem as you and i can solve it with this solution.

slm
  • 7,355
  • 16
  • 54
  • 72
Tarik Nasser
  • 1
0

I had the problem that you write. The test I ran was from perl. Put this line into the file /etc/nagios/nrpe.cfg to make it work.

command [check_memory] = /usr/bin/perl /usr/lib64/nagios/plugins/check_memory -w 75-c 90
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
user224319
  • 1
0

There's a really nice article which covers the whole NRPE agent installation and configuration with many check_commands examples, I use this article when ever I need to install NRPE on a new server. More than that, in the end of the page you can find a cool script which automatically installs and configures NRPE for you (based on variables you set), the article can be found: Here

Itai Ganot
  • 10,424
  • 27
  • 88
  • 143
0

This usually happens when NRPE server is started with the user nrpe, instead of nagios.

Changing the nrpe_user value into nagios in the /etc/nagios/nrpe.cfg file should solve your problem.

The nrpe_group can be changed too if needed.

Umut Uzun
  • 101
  • 3
0

One other thing to check is that if your command is using sudo -u <another user> to run the command, the libexec directory (and directories above it) must be readable by the user being sudoed to.

For example if your command is:

command[check_tomcat]=sudo -u tomcat /usr/local/nagios/libexec/check_tomcat ...

The tomcat user must be able to access that file.

One way of fixing this would be:

chmod 0775 /usr/local/nagios/
chmod 0755 /usr/local/nagios/libexec

Replacing the last part with wherever your executables live

Mitch
  • 131
  • 1
  • 1
  • 5
0

I had the same issue and I manage to solve it by killing the nagios process (on the monitored machine):

ps -ef | grep nagios
kill -9 [NagiosProcessNumber]
/etc/init.d/nagios-nrpe-server start

All went fine after that.

Colt
  • 1,939
  • 6
  • 20
  • 25
user428879
  • 11
  • 1
0

Just had this issue on FreeBSD. After banging my head against a wall for an hour I realised the issue was that the /usr/local/nagios/etc/nrpe.cfg was pointing at the wrong location for sudo.

To find the correct location to point to for the sudo command run:

# whereis sudo

I then changed the command_prefix in nrpe.cfg from:

command_prefix=/usr/local/sudo

to:

command_prefix=/usr/local/bin/sudo

Then ran service nrpe restart and the problem was solved.

Could be a similar issue on other operating systems, just another thing to check if you have checked all of the other possible permissions issues and you still hit this problem.

Graeme
  • 1
-1

Missing Nagios plugins on the nrpe client.

Do not use yum install nagios-plugins ( nagios-plugins-2.0.3-1.el6.x86_64). It does not install all the plugins. Download nagios-plugins-1.4.11.tar.gz and follow instruction in this document.

http://www.thegeekstuff.com/2008/06/how-to-monitor-remote-linux-host-using-nagios-30/

Jim
  • 1
-2

I had this problem and I solved disabling the selinux

setenforce 0

Paulo Azedo
  • 1