Monitoring bespoke software with Zenoss

Question

We've got a lot of back-end applications that we need to monitor the performance of (metrics such as orders waiting to be processed, time since last run, etc). Currently, this is done by an in-house watchdog application that fires out emails whenever a threshold is exceeded, but there's no way to acknowledge an issue and squelch these alerts.

Rather than build our own complete alerting system, we'd like to tie in to the Zenoss installation we use to monitor our servers. I've found a few articles on creating events programmatically, but I'd rather Zenoss itself monitors the values that the current watchdog app is looking at (so we get the benefits of graphing and history as well).

Is it possible, then, to programmatically provide a data feed (rather than an event) to Zenoss? Or is there another way to go about this?

Answer 1

The cleanest solution, IMHO, is to let the app provide information via SNMP.

Then you can monitor it using any software that speaks SNMP, including, but not limited to Zenoss.

There are various ways to achieve this.

I have done it myself for some custom apps:

The apps were running on a Linux server, which already ran the Net-SNMP daemon. So I wrote a plugin for Net-SNMP (just a small Perl script), which queried the values from the app and reported it to Net-SNMP.

I used the exec mechanism of Net-SNMP to run it ( http://net-snmp.sourceforge.net/docs/man/snmpd.conf.html#lbAZ ). Basically you just put

exec  [MIBOID] NAME PROG ARGS

into the snmpd.conf. Then Net-SNMP will invoke your script, and report its results back via SNMP. Your script just needs to print the result(s) on stdout (one per line if it reports multiple values), otherwise it does not have to do anything specific. Note: Using exec is now deprecated in favor of extend, but the principle is the same.

There are other, more powerful extension mechanisms (you can write a plugin in Perl, or a dynamic module in C, ...), but this is a good place to start.

Other SNMP daemons will have similar extension mechanism, it just depends on what you are currently using on the server that your app runs on.

Also, there is a special protocol called AgentX to allow an app to serve as a "sub-agent" (that is, to report data to the main SNMP daemon on a system). You could even implement a subagent in your app.

In short, there are many ways to accomplish data reporting via SNMP; just choose the simplest solution, then extend it when you need to. At any rate, that way you get an extensible, standard-based solution, instead of ad-hoc emails.

Edit:

To do this under MS Windows:

One possiblity (there may be others, don't know) is to install Net-SNMP under Windows (they have a Windows version). You actually have two options:

1. Keep the Windows SNMP service running, and install Net-SNMP alongside it
2. Disable the Windows SNMP service, and run Net-SNMP instead

Under option 1, you will need to let the Windows SNMP service use a non-standard port, and have the Net-SNMP agent proxy requests to it. Under option 2, the Net-SNMP agent will directly load the DLLs which the Windows agent would use if it ran. Thus in both cases, you should still get the MS-specific information the Windows agent provides. Both approaches have some drawbacks; see the README.win32 for details.

Once you have Net-SNMP running, you can use all its extension functionality, just like under Linux (see above).

You can also extend the Windows SNMP agent directly (without using Net-SNMP), but I don't know much about that. There is a "Windows SNMP API", which is apparently one way: http://msdn.microsoft.com/en-us/library/aa379207%28v=vs.85%29.aspx

Answer 2

I wrote a paper a couple of your back now discussing different options for "process monitoring" with Zenoss - http://community.zenoss.org/docs/DOC-3537 .

Not much has changed since then other than that the built-in Zenoss process monitoring is a little less quirky and more reliable. It discusses various ways including using the net-snmp daemon (which incidentally is also available for Windows). It also discusses running scripts using ssh. Either way, you can graph collected data, in addition to alerting on thresholds.

Cheers, Jane