0

I have an environment that generates a frankly ludicrous amount of Syslog traffic - this is mostly due to a culture of leaving debug-level logging on in production applications (and changing this tendency is unfortunately off the table and way above my pay grade).

I've already had to move away from UDP due to the sheer volume causing packets to be lost. A smaller centralized syslog environment (many senders, a handful of receivers) is being quickly overrun, with either the individual syslog servers or their backing storage failing multiple times under load.

I'm now in a position to rearchitect this.

What I'm thinking is something like having syslog-ng on my clients (already standard per config management enforcement), sending out the messages formatted as JSON, via an HAProxy cluster, with the messages coming to rest on an Elasticsearch cluster. As a bonus, both HAProxy and Elasticsearch are independently, horizontally scalable with future needs.

I don't immediately foresee any problems with this, but I'd also like a second opinion. Is this a viable plan? Are there better options? Are there any footguns I should be looking for?

Note: Just to be safe, I want to clarify this question is about sending TCP syslog traffic -to- HAProxy to be load balanced, not HAProxy's own logging of its traffic or status

Mikey T.K.
  • 1,367
  • 2
  • 15
  • 29

1 Answers1

0

Graylog and ELK are log management projects that also can horizontally scale, and happen to use Elasticsearch storage. Inputs can be load balanced however you like, how to articles exist for HAProxy, naturally.

However, you haven't shared many details that would allow commentary on your architecture. Why is your storage failing (you should probably fix that first)? What is the messages per second rate that's giving you problems? How many nodes were you considering? Do you really Each of these could be their own question.

John Mahowald
  • 30,009
  • 1
  • 17
  • 32