2

I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing. I see 2 variants:

    1. write logs to local log files for each servers
    2. rename logs for template "DD.MM.YYYY HH" in each servers and send logs to stat server over ssh, for example
    3. send signal for reload logs to nginx on each server
    1. use syslog transport ( or other - which transcport may autorename file for template "DD.MM.YYYY HH" ? ) for send logs to stat server
    2. use hadoop or temporary dir in memory on stat server for writings logs from all servers ( which variant more productive in this case ?)

  3. Can you suggest something else?

Sven
  • 97,248
  • 13
  • 177
  • 225
Bdfy
  • 181
  • 1
  • 1
  • 3

2 Answers2

2

Take a look at elasticsearch + logstash + kibana

Logstash can parse your log files and extract information from them using filters. It can store them in Elasticsearch, which you can query via Kibana. Have a look at the videos on their site to get a feel about what it can do.

Andrei Nistor
  • 176
  • 3
0

Your first option could be done via loghost (basically your option 2) You can send your syslog message via udp to $logserver . Then you would split the logs there based on server and have the "exact" timestamps of the logs you did send.

The renaming of logs would be not necessary on the nginx servers, other than the usual logrotate you most likely need with that much logging anyway.

Its best not to script too much if the tools are available already, no need to reinvent the wheel here.

Option 2 should be pretty close to what you want to achieve.

Depending on the security of the network(s) the servers are in: either transport directly via "normal" syslog procedure or make f.e. some vpn connection to the loghost.

Dennis Nolte
  • 2,848
  • 4
  • 26
  • 36