1

Summary

How can I use a remote admin tool (puppet or chef) combined with a sync tool (e.g. rsync) and scheduler (cron) to send log files from several servers to a central server?

Context

Our sysadmins have (rightly) tightened up developer access to production servers. One consequence: developers in particular will no longer have easy access to log files when supporting customer problems.

Goal: Send Files to Common Log Server

I'd like to set up automatic copying of logs to a central "log server". Developers/support would have read access to the 'log server' filesystem when supporting problems.

Here's an example of our filesystem structure. One directory per customer. Within this, one directory per application. Each application has different sub-directory structure and log file names.

Server A
d:/
  Instances/
     CustomerFoo  
         App 101/
            logs
               server.log
               access.log
         App 201/
            logs
               stdout.log
               jdbc.log
               timing.log
     CustomerBar/
         App 101/
          ....
         App 301/
             logs/
                monitor.log

These would go to a central server with a structure like this. (Customer name is unique within the enterprise so we will not have any collision)

LogServer
d:/
  logfiles/
     CustomerFoo  
         App 101/
            logs/                   
         App 201/
            logs/
               ...
     CustomerBar/
         App 101/
          ....
         App 301/
             logs/

Desired outcome

  • I'd like a combination of a remote admin tool (puppet or chef) combined with a sync tool (rsync) and scheduler (cron) to do periodic syncing
  • Easy remote administration for multiple boxes. i.e. edit the 'config file' in one place and push it out to all the servers.

Environment

  • Windows Server 2008
  • Windows shop. However open to installing cygwin.
  • I personally am familiar with rsync and cron, but neither puppet nor chef. fwiw.

Thanks in advance!

user50460
  • 315
  • 1
  • 2
  • 8
  • 3
    Why not use a proper log shipping app to get your logs in realtime? – EEAA Jan 28 '14 at 17:04
  • 2
    Right - what's wrong with using, er, syslog? – mfinni Jan 28 '14 at 17:05
  • 1
    Additionally, if you need access to these logs to do your job, there's no reason your access to them should be restricted. – EEAA Jan 28 '14 at 17:05
  • what mfinni said – Chopper3 Jan 28 '14 at 17:07
  • Maybe consider something like [logstash](http://logstash.net). – cjungel Jan 28 '14 at 17:21
  • @EEAA : "proper log shipping" app. Please suggest one. Also, for some reason I only seee the term "log shipping" in the context of database transaction logs. Hence I used log copying/sending – user50460 Jan 29 '14 at 02:20
  • @mfinni re syslog. Tell me more, especially in a Windows environment. That said, these windows sysadmins look suspiciously on unixy stuff. – user50460 Jan 29 '14 at 02:22
  • Your application needs to be able to speak syslog, preferably to a remote syslog server. Looks like you're using Java? Hope you're using Log4J, because that will speak syslog. Then you point them at your syslog server. there are commercial and free syslog servers available for Windows. – mfinni Jan 29 '14 at 14:13

1 Answers1

0

I suggest syslog. It's a (the) standard for what you want.

Your application needs to be able to speak syslog, preferably to a remote syslog server. Looks like you're using Java? Hope you're using Log4J, because that will speak syslog. Then you point them at your syslog server. there are commercial and free syslog servers available for Windows.

Alternatively, you could write everything from your application to the Windows Event Log, and then use event log subscriptions/collection to get those to another Windows machine that you're allowed access to.

Doing this with only plain-text logs, when you have multiple nodes, is tough to manage, as you're finding out. Of course, you could just request read-only access to the directories where the logs currently live.

mfinni
  • 35,711
  • 3
  • 50
  • 86