I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing. I see 2 variants:
- write logs to local log files for each servers
- rename logs for template "DD.MM.YYYY HH" in each servers and send logs to stat server over ssh, for example
- send signal for reload logs to nginx on each server
- use syslog transport ( or other - which transcport may autorename file for template "DD.MM.YYYY HH" ? ) for send logs to stat server
- use hadoop or temporary dir in memory on stat server for writings logs from all servers ( which variant more productive in this case ?)
Can you suggest something else?