5

What are some good tools for monitoring logs on a system? I am currently using logsentry, which basically just runs grep on logs, ignores regexps I tell it to ignore, and mails me everything else, but I'd like to find something a little 'smarter', (ie can understand what a Postfix log is beyond just being a series of lines to grep on, things like that - contextual knowledge).

On services, not running anything particularly exotic - apache, memcached, postgres, postfix, openssh, squid, bind, mailman. About the strangest thing running is monotone, which I would not expect any log analyzer supports (thus, it would be nice if it is easy to extend with understanding of additional services).

Something that can parse and summarize information in hourly or daily reports (eg, 15 404s reported in this apache log file, 3 logins by this user from IPs x, y, and z, 5 mails sent by this user and 150 received by that other user, disk X is 90% full, 3000 failed ssh attempts from these random IPs, etc) would be ideal.

I've looked through several lists of log monitoring tools for Unix but most of them seem to boil down to tail -f or grep. I'm hoping there is something out there that I've overlooked.

Jack Lloyd
  • 305
  • 1
  • 3
  • 9

4 Answers4

6

I think of monitoring in two types. There is monitoring the current state and keeping a history of that on the monitoring server, and also there is monitoring history (logs).

Some of the information you listed isn't really meant to be monitored via logs. Disk space for instance, would probably be better monitored by a plug-in that calls the df command or something like that. So I would consider this monitoring the current state, even if you keep a record of it.

I like Splunk for log monitoring, but it is expensive if you need it to do things the free version doesn't. For monitoring things like disk space, if a process is running, cpu usage, etc, personally I like Nagios. The two pair well together I think.

You might find something that does both of these well, but I wouldn't try to get one type to do the other (ie Nagios to monitor logs), it will probably be a little hackish.

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
4

You can use Octopussy for all the reporting stuff you asked. (except the '90% disk full' as Kyle said probably better monitored by System monitors like Nagios, Zabbix,...)

Octopussy is a log management solution to:

  • Search in your logs
  • Raise Alerts from logs
  • Generate Reports from logs

With Octopussy, you can 'easily' create reports on anything you want from any logs. (may be not so 'easy' because you need some work to define logs patterns & reports, but really powerfull after that first work)

Disclaimer: I work on Octopussy.

sebthebert
  • 1,224
  • 8
  • 21
2

What you want is OSSEC, seriously. It puts logsentry, logwatch and similars down easily.

It does exactly that by default, detecting web scans, brute forces and so many issues using log analysis (or log-based intrusion detection ,like they call it).

Try it out, it is the most easy tool to install (just run their install.sh script) and enjoy.

Link: http://www.ossec.net

sucuri
  • 2,817
  • 1
  • 22
  • 22
0

Logwatch, which is installed and active, I believe, on both Debian and Redhat, does many of the reports you are looking for. I expect there is lots of overlap with what you get out of logsentry. For some reason, configs of logwatch on Redhat produce a lot of volume (esp samba log), making it more useless, contrasted with tighter reports of logwatch from Debian.

Personally, I put more into watching just the things that are likely to go critical. I have cron jobs to test if the RAID devices see all healthy disks and email me if not. I use cacti to graph the active and total queue on the MX and SMTP servers (by shell script queried over snmp). Likewise I have daily stats on postfix logs which are graphed in cacti. Denyhosts blocks excessive ssh or other service auth failures and emails root. Even the server room temperature can be monitored by getting the temperature of an APC UPS (use apsupsd) and plotting that in cacti. Cacti will also graph disk usage, server load, etc. The great thing about graphs is that they will show you trends over weeks or years which you might not perceive otherwise. For example, how much spam is tagged or disk space trends. This allows for future planning.

labradort
  • 1,169
  • 1
  • 8
  • 20