8

We have several applications which are generating their own plain-text log files, which I would like to forward to a remote syslog server for centralized logging. I do not have access to root on these machines, nor can I reconfigure syslog to redirect output to a remote machine.

I've found some solutions online, but they're mostly people's homemade bash scripts, and I'm looking for something more robust that is suitable for implementation in a potentially high-volume production environment.

Preferably something designed with an eye for a small footprint, background daemon that keeps running, that can keep up with a lot of lines, etc. - What solutions are currently available?

Avery Payne
  • 14,326
  • 1
  • 48
  • 87
Michael Martinez
  • 2,543
  • 3
  • 20
  • 31
  • 4
    Have you looked at the [text file input module](http://www.rsyslog.com/doc/imfile.html) for rsyslog? –  Oct 22 '13 at 22:31
  • @yoonix: No I haven't, but I'm going to :) – Michael Martinez Oct 22 '13 at 22:34
  • 3
    Uhm, syslog can send to remote syslog servers. Configure your local syslog to send to a remote server. Then access your local syslog via the standard syslog calls or by using logger or something. – Zoredache Oct 22 '13 at 22:41
  • the log files in question are not generated by syslog. view them simply as text files that are being generated directly from an application. – Michael Martinez Oct 22 '13 at 22:46
  • yoonix: imfile module looks interesting, but I'm putting it as a second or third choice, because it requires modification to rsylog.conf, and (a) I don't know if the server is actually running rsyslog, (b) neither me nor my team members have root or sudoers on the box, so ideally we need a solution we can test as a regular user. I'm currently looking at "swatch" as a possible contender. – Michael Martinez Oct 22 '13 at 22:50
  • 4
    Why don't you write your log files to a named pipe and have a daemon listening that sends them on their way http://serverfault.com/questions/189477/rotate-logs-of-a-dumb-non-interactive-application – user9517 Oct 23 '13 at 20:37
  • @lain: I thought about that, but it would still require modification of the node.js app. A less intrusive solution would be better. – Michael Martinez Oct 23 '13 at 20:39
  • @MichaelMartinez: Why? just modify what it's writing to to be a named pipe it doesn't need to know what is. – user9517 Oct 23 '13 at 20:42
  • @lain: modifying the app code is a last resort. Otherwise, your solution would be a possibility. – Michael Martinez Oct 23 '13 at 20:48
  • 3
    You shouldn't have to modify the app just put a named pipe of the same name as the log file the app is writing to in place . – user9517 Oct 23 '13 at 20:51
  • @lain. that would require root, which I don't have. – Michael Martinez Oct 29 '13 at 19:14
  • I've updated the question to include the missing requirements. Without those requirements present, you're going to get a lot of "dude, why are you doing that" instead of "dude, why can't you ask for sudo so you can `vi /etc/rsyslog.conf`?" – Avery Payne Mar 12 '14 at 19:02

5 Answers5

14

You've already rejected "other people's bash scripts", but this is a pretty common solution -- some creative use of the logger command can follow a file and send its contents elsewhere.
I personally wouldn't do this in a production environment though.

A better option which requires less scripting hackery is using rsyslogd and the text file input module like yoonix mentioned -- This is a pretty decent solution though there is some potential for lost lines during a file rotation, and if you're on a Linux system with rsyslog as your syslog daemon there's not much additional work required.

syslog-ng also supports a file input source with functionality similar to rsyslog's.

IMHO the best solution - albeit one which requires modifying the application generating these logs - is to log to syslog directly. You don't want to be going through intermediary steps, files, etc. -- syslog is the SYStem LOGger, and things that write logs on a Unix platform should be sending them to syslog.
Implementation of this is, unfortunately, left as an exercise for the reader (and application developer) and may not be possible if your developers are nonexistent, lazy, or incompetent....

Community
  • 1
voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • The OP says quite clearly that he doesn't even know if it's running rsyslogd and has no admin rights so really this is a useless answer. – user9517 Oct 23 '13 at 20:44
  • If I can run syslog-ng (or rsyslogd) as a non-privileged user, then this will be my first choice. The host is already running rsyslogd as root, so I'll have to see if it's possible to run another copy simultaneously as a different user, possibly tied to a different port. Ideally, yes, I would like to modify the app to write directly to syslog. Unfortunately this a last option. I don't know who that group is, and since I work at one of the largest internet companies in the world, just finding them is possibly a multi-day ordeal. Otherwise, I agree with you 100%. – Michael Martinez Oct 23 '13 at 20:47
  • 7
    @MichaelMartinez You would modify the `rsyslog` configuration currently running on the system. You should ***NOT*** be running two syslog daemons. Not to be rude, but you need to stop trying to do it wrong*: Every proper solution to this scenario requires administrative (root) actions on the server or modification of the app. You're going to have to face that reality and deal with whatever group within your organization has root on the systems in question, otherwise this question *is* off-topic (you're trying to circumvent your organization's policies).... – voretaq7 Oct 23 '13 at 20:57
  • voretaq7 Actually I believe the swatch solution may work, without requiring administrative privilege. One question for you: how do know this organization has a policy against what I'm doing? You're making assumptions. In fact the modus operandi around here encourages exactly what I'm doing. In the words of my boss on this very task: "WE aren't allowed sudoers. We need to figure out another way for now." – Michael Martinez Oct 23 '13 at 22:06
  • 5
    @Michael All this tells us is that someone is trying to force the wrong team to implement the fix. – Andrew B Oct 23 '13 at 22:49
  • @Andrew Such as it may be, one has to work with it. – Michael Martinez Oct 25 '13 at 17:37
  • 4
    @MichaelMartinez imho, that sounds like a pretty quick route to crippling levels of technical debt. – Sirex Oct 29 '13 at 19:11
  • 2
    @Sirex. Be it what it may, it's the way of things. I work at an organization that employs 10s of thousands of people, most of whom are technical (engineers, dev, ops, etc) – Michael Martinez Oct 29 '13 at 19:14
  • 5
    I guess. Generally i've found long term that there's no medals in winning self-inflicted battles. When the technical debt gets to the point it impacts the business ironically the people who diligently worked hard to avoid the elephant in the room tend to end up carrying the can, in my experience. So i'd say cover your ass and get someone to agree in writing the downsides of this. – Sirex Oct 29 '13 at 19:29
  • 1
    @MichaelMartinez RE: "10s of thousands of people", my [answer](http://meta.serverfault.com/a/5911/152073) to your meta question still applies; you are a professional, what your company is asking of you is not. Emphasizing the size of your organization is fairly meaningless here; many of us have been around the block long enough to know that the size or (self) importance of a business are no barrier to management mistakes. – Andrew B Nov 05 '13 at 21:59
  • @AndrewB To me, this is pretty meaningless. I'm not a manager, nor do I have the desire to go around trying to change the way things are done. I simply want to get the job done and implement a solution. A lot of the responses I see here are "heads in the sand" kind of things. – Michael Martinez Nov 07 '13 at 00:12
  • 1
    @Michael If you just want an answer and don't care about all this dithering, migrate it to unix.SE as suggested in the meta answer. You won't have to deal with this there. – Andrew B Nov 07 '13 at 00:44
7

You could use logstash with the file input and syslog output.

For example, create a configuration with the file (or files) you want to monitor and your syslog server info.

file-to-syslog.conf:

input { file { path => "/var/log/kern.log" } }
output {
    syslog {
        facility => "kernel"
        host => "syslog.example.com"
        port => 514
        severity => "informational"
    }
}

The start up logstash with

java -jar logstash-1.2.2-flatjar.jar agent -f file-to-syslog.conf
sciurus
  • 12,493
  • 2
  • 30
  • 49
  • +1. if using rsyslog's file input isnt an option, logstash is the next best thing. In many ways it's better in the long run. – Sirex Oct 29 '13 at 19:12
  • I'm not familiar with this. If it does what I need, it would have saved me the trouble of hacking coreutils and util-linux. – Michael Martinez Oct 29 '13 at 19:12
  • yeah, the config is going to look a bit like this: http://pastebin.com/xeC9hxD3 – Sirex Oct 29 '13 at 19:16
  • looks like a very cool tool, but definitely overkill for what I need here. logstash is its own service, with web interface, requires java, etc. I'll continue to use my filelogger which is lightweight, small footprint, optimized for performance. ... But, thanks for suggesting logstash because I can see a need for it in other situations in the future! – Michael Martinez Oct 29 '13 at 19:18
  • yeah its a jar packed jruby tool. The gui is actually kibana which is packaged into it for easy but is actually a seperate project, so it's not needed just for parsing messages. It's basically a swiss army knife of logging. You define inputs and outputs and in the middle you can optionally grok the logs, which gives them context. -- IT's likely overkill for you unless you also want to use elasticsearch on your log data. – Sirex Oct 29 '13 at 19:23
  • I agree the logstash website gives a misleading first impression of the project. I edited my answer to show how straightforward using logstash could be. – sciurus Oct 29 '13 at 19:49
4

I hacked together tail.c and logger.c into a single, small footprint compiled program (binary) that is lightweight, fast and stable. As long as it has read access to the log file(s), then it works without needing root privilege.

I also made a couple improvements to the native logger and added a new (optional) capability of inserting a text string at the beginning of each log line before it gets sent to the log server. The result is a program that can be run by itself, without needing to use shell pipes (i.e. don't need to tail logfile | logger). It will run forever until explicitly killed or it encounters an error writing to the network socket. It even continues to run if the log file is rotated or even disappears (it will just continue to look to see if the file reappears.)

It's easy to use: just give it one or more log file(s) to monitor, and each time a new line gets written to the file, it will send a copy of that line to the local or remote syslog server you specify. Plus the extra text string if you use that option.

I actually finished the program back in December, but was waiting for Yahoo to take copyright and make it available, which they've now done. (I wrote it as part of my job at Yahoo).

filelogger program information and download link:

slm
  • 7,355
  • 16
  • 54
  • 72
Michael Martinez
  • 2,543
  • 3
  • 20
  • 31
1

There are a number of ways to tackle this. But the very, very first thing you should do is: forward the logs using syslog itself.

Syslog (and many replacements for syslog) have built-in facilities to forward logging to another syslog server at a different address. You can easily do so by changing the configuration file and appending the address to forward the facility to. For instance, adding this line to:

*.*    @192.168.1.1

...would forward all facilities to the machine at 192.168.1.1, which (hopefully) has the service running. The example I give here is for rsyslog, which is the stock syslog server on Debian, although it should work for many others. Consult the documentation for your implementation of syslog with man syslog and see what it says about "forwarding".

The remote syslog server can be anything you like. There are even products, like Splunk, which happily aggregate these logs into a single view with a web dashboard, search, event-driven notifications, etc. etc. You can see more here: http://www.splunk.com/ If that doesn't meet your needs, you can use something else. There are even syslog servers that will dump to a SQL database!

Sure, you could write your own script/program/service to do this for you, but why re-invent the wheel when it's both done for you and already given to you?

Edit: So I went back and re-read the question, and noticed several comments. It sounds like:

  1. you want to aggregate your application logs
  2. you don't have access to root
  3. your application(s) just dump text somewhere
  4. your application(s) doesn't know how to write to the local syslog
  5. you don't have control over your application source code

So let's address each one in sequence:

  1. syslog was meant to aggregate logs together. You can use anything you like, but there is a reason why it's been around for a long time. It's well-tested, well-debugged, well-documented, well-known, and for most *nix platforms nearly universally supported in one flavor or another.
  2. we don't need access to root to set up logging. We only need access to the syslog API. root is not a requirement to write to the syslog; if this were the case, then all of those services that drop privileges would be unable to write diagnostics to the log files.

  3. Re: text dumps, this is normal. however, you should be able to use a subshell to pipe the output of STDERR and STDOUT to a program that calls the syslog API. This isn't rocket science, it's far from being brittle, and it's well documented. In fact, it's one of the reasons that output redirection even exists. A simple command that could be tossed into a single shell script would be:

    ( my-application 2>&1 | my-syslog-shunt ) &

  4. if you have the ability to alter your application's source code, you should write a shunt into it to dump the text output to syslog instead of a plain text file. This shouldn't be too hard; all you do is take the lines you would output, and wrap them with a call. However....

  5. you might not have access to the source code at all, so you can't do this. Which means something like #3 above would work fine.

Avery Payne
  • 14,326
  • 1
  • 48
  • 87
  • two reasons: (1) simply because, as already mentioned, didn't have root or sudo on the boxes in question. (2) "logger" itself can forward to remote server, but has a 400-character limit per log line, which is not appropriate for Apache logs. Anyway, I already put together a custom solution which does exactly what I needed (and improves "logger" too). See my answer here for "filelogger" – Michael Martinez Mar 12 '14 at 18:47
  • 4. Syslog isn't just a file stream I can open and write text to. The shunt I write would have to open up a socket to the UDP port that syslog listens on? – Noumenon Sep 08 '18 at 18:50
  • 1
    @Noumenon, I'm not entirely clear on your intent, but I'm assuming you want to pipe program output into the system log, which can be done with the logger command. https://linux.die.net/man/1/logger – Avery Payne Sep 11 '18 at 23:00
  • @AveryPayne So like `Runtime.exec("logger ..."`) OK, thanks. – Noumenon Sep 11 '18 at 23:08
0

I'm answering my own question.

swatch might have worked, but I was unable to get perl's Sys::Syslog module to work on the host, and the /usr/bin/logger that installed on the host does not support logging to remote server (util-linux-ng-2.17.2).

So, the first thing I did was to download the source code for util-linux-2.20.1 for which the logger program does support remote logging. Upon testing, it became apparent there is a limit imposed on the number of characters allowed on the log line. Digging into the source code I found a hard coded 400-character limit. (If you don't believe me, run "strings /usr/bin/logger | grep 400" on any Linux system).

This limit is not acceptable for apache-type of logging (including nodejs), so I modified the code and increased the limit to 4096. While I was at it, I also added a new command-line option which allows one to insert an optional text string at the beginning of each log line. I did this because the nodejs logs do not include the hostname as one would might see in apache.

At this point, I could run a shell script with "tail -F -n 0 [logfile] | ./modified_logger ...." and it worked. But I had some concerns about running this from supervise (daemontools) or even in the background, because if one or the other sides of the pipe terminates, then there is the risk the whole pipe will terminate. I also had concerns (albeit untested) about performance.

so I decided to combine the tail functionality with the logger functionality into a single executable binary that would bypass the need to use Unix pipes or external programs. I did this by hacking tail.c from gnu coreutils and incorporating what I need into the modified logger program.

The result is a new binary (117k size) which I'm calling "filelogger" and which continously monitors one or more files and logs each new line to a local or remote syslog, either via UDP or TCP. It works like a charm. I was able to do a little benchmarking and it logs about 17,000 lines (1.8MB) in about 3 seconds across subnets with a vlan and a couple physical switches between them, to a remote server running syslog-ng.

to run the program you do something like the following (either in the foreground, background, or supervised with daemontools):

./filelogger -t 'access' -d -p local1.info -n [remote loghost] -u /tmp/ignored -a $(hostname) /tmp/myfile1 /tmp/myfile2 ...

/tmp/myfile1 and /tmp/myfile2 are the files being monitored.

The "-a" is the new option I added. In this case I insert the local hostname at the beginning of each log line.

This solutions was exactly the type of solution I was looking for when I asked the question and, as it turned out, did not exist until I made it myself. :)

Michael Martinez
  • 2,543
  • 3
  • 20
  • 31
  • I'll probably make this available on sourceforge at some point. It's advantages are that its a very small footprint, lightweight, easy to use, and optimized for performance. Once message text is read, all processing is done in memory buffer then transferred directly to socket. – Michael Martinez Oct 29 '13 at 19:21
  • 1
    http://xkcd.com/763/ – user9517 Oct 30 '13 at 19:35
  • 4
    I'm trying not to be harsh, but I *am* going to be blunt: This solution didn't exist because it's terrible. Rather than interfacing with the other groups in your organization and implementing a sane, standard solution you've thrown a hack in place with totally unsupported code that *you* now need to test/debug/maintain going forward. You ignored easily 50+ years of combined experience telling you "Don't do that" -- I hope for your sake this doesn't blow up in your face, but you are absolutely, unquestionably *Doing It Wrong* here... – voretaq7 Oct 30 '13 at 19:55
  • 1
    yeah. right.... This is how open source moves forward, dude. If everyone did it your way, there'd be no progress. How do you think GNU, Linux, and everything that's based on it came about? People doing exactly the sort of thing I did here. If it makes you feel better, I do intend to my code into our package management system, where everyone here at the organization is free to use it, deploy it, and improve upon it, if they so desire. – Michael Martinez Oct 30 '13 at 21:32
  • And FYi, it's not a terrible solution. To the contrary it's a very useful tool. When I was searching online for solutions last week, I came across other people asking where they could find this exact functionality. – Michael Martinez Oct 30 '13 at 21:47
  • You do realize that syslog has built-in forwarding, right? That you can forward log facilities to other syslog servers by changing the configuration file, right? – Avery Payne Mar 12 '14 at 18:35
  • yeah, but configuring syslog requires root, which was not an option. – Michael Martinez Mar 12 '14 at 18:51
  • I updated the question to better reflect the requirements. Without those requirements, the answers you are getting make sense...because that's how just about everyone else addresses the issue. With the requirements, the answers will change. – Avery Payne Mar 12 '14 at 19:03