2

We have a small project we are working on part-time that runs on Nginx/MongoDB on Ubuntu 10.04 LTS Server. We'd like to be able to see reports on things like server load, requests/sec, response time, DB load, DB response time, etc. Is there an open source or free (as in beer) tool that can parse such logs and provide a real-time report?

I looked into Splunk briefly, but I wanted to see if there are any others that are highly recommended.

Alienfluid
  • 123
  • 1
  • 3
  • How are you going to get server load out of a parsed log? – womble Aug 05 '11 at 03:56
  • If all you're looking for is load, there's a lot of tools out there for that, which aren't necessarily log-related. @womble, he's probably referring to the splunk *nix module, which digests system load data into text log entries on specific intervals. – Shane Madden Aug 05 '11 at 04:11
  • This sounds more like a job for Cacti than for log parsing per se. – anastrophe Aug 05 '11 at 04:48

4 Answers4

3

Most of the metrics you would like to see are accessible through the respective applications statistics, which is much more effective than log analysis. And there are several free projects aimed to collect such metrics, build graphs, check margins, send warnings, etc. All of them are designed to be easily extensible, so that a lot of plugins for collecting metrics exist and you can write your own. The names of the main projects I remember of are Munin, Cacti and Graphite, but in fact there are more.

Munin is one of the oldest and has a lot of plugins even in the standard package. From your list:

  • server load Munin has plugins to show load average (LA), iostat, vmstat
  • Nginx Munin has "nginx_request" (req/sec) and "nginx_status" (number of connections in a specific status, i.e. active, reading, writing)
  • response time As far as I know, there is no such plugin. See this question for more information
  • MongoDB MongoDB team provides several Munin plugins at GitHub
Community
  • 1
Alexander Azarov
  • 3,510
  • 20
  • 19
0

You can do it with a shell script and push the result to Ganglia gmetric or Nagios + PNP4Nagios.

quanta
  • 50,327
  • 19
  • 152
  • 213
0

AFAIK there's nothing available for measuring requests/sec and response times. An answer to what tools you might use to capture and report on these metrics alone would fill a large book!

For Mongo, the choice is bit simpler. Assuming you don't want to roll-your-own from scratch, then have a look at Server Desnity's (hosting company) monitoring toolkit

symcbean
  • 19,931
  • 1
  • 29
  • 49
0

You might want to take a look at our NGINX sFlow module along with sFlowTrend.

NGINX sFlow module

sFlowTrend adds web server monitoring

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
Peter Phaal
  • 1
  • It, ummm, might be wise to disclose your association with sFlow, and [read this section of the FAQ](http://serverfault.com/faq#promotion). – pjmorse Oct 11 '12 at 15:28