1

the main requirement I have is to be able to view and do simple searches on logs combined from multiple machines. However, I would like the solution to have minimal impact on the rest of the (core) system. I don't have any real-time requirements, the process can be asynchronous.

Initially syslog seemed like a good option, but what if the syslog server dies? In the worst case the core system's user see errors, in the best case some logs are lost.

So I started looking around and found Logstash (http://logstash.net/). Currently my idea is:

  • on each server (that's running the system's core components) there is a Logstash agent running
  • the agent monitores log files and sends them to an ElasticSearch cluster
  • there's another server with the Logstash UI

That way:

  • there's no single point of failure
  • even if the ES cluster dies, only the agents will be affected - the application will still happily write logs to files
  • after ES comes back, the agent will (hopefully) catch up and send all the pending logs (is Logstash smart enough to do that?)

Do you think this will work? Or maybe you could recommend another solution?

adamw
  • 135
  • 1
  • 5

2 Answers2

1

Rsyslog has some of the functionality that you are describing and the project also has plenty of documentation about reliable message forwarding.

In short, with rsyslog you can use the RELP protocol for reliable syslog message forwarding and then you don't have to worry about message loss. And you also have the option of configuring local spool files, where rsyslog will buffer messages in case the remote server is down. Once the remote comes back up, your agent will catch up.

You also have the option of configuring rsyslog to write to a relational database, and then you can make the database as redundant as you want (I personally find a syslog server easier to cluster).

chutz
  • 7,569
  • 1
  • 28
  • 57
0

This question is most likely going to be closed as off topic, please see the FAQ.

Regardless, syslog (or any syslog based system) should work just fine, after all if you are concerned about losing logs then make sure to backup the syslog server as part of your normal DR scenario. It's a pretty simple task/request.

Brent Pabst
  • 6,059
  • 2
  • 23
  • 36
  • I took a look at the FAQ, seems on-topic to me :). – adamw Nov 06 '12 at 21:45
  • Sorry, but no it is not. `Or maybe you could recommend another solution?` is a product and solution solicitation, check the FAQ again, that is not allowed on the site. I'm glad you found an answer but please double check to make sure your questions are about a specific problem or question in the future. – Brent Pabst Nov 06 '12 at 21:58
  • Well, I wasn't looking for a product, sorry maybe the wording wasn't best, I'm not english-native :). But I would say that most of the questions look for "solutions" to problems, here looking for a fault-tolerant log collection. Anyway, answered, thanks for the help :). – adamw Nov 07 '12 at 11:28
  • Yes, just be careful about how you word your questions in the future to avoid `soliciting` recommendations or potential solutions. Instead, identify one and then ask questions about it. If you do that, much less chance of someone (like me) voting to close the question. – Brent Pabst Nov 07 '12 at 13:40