11

I have just setup a Graylog2 server and I am looking to send all logs from my main server to the graylog server. I have enabled logging for the main server and am sending logs to my graylog server by adding *.* @logs.example.com:1337 to /etc/rsyslog.conf.

What I want is to have Graylog2 collect all my Apache logs, system logs (for SSH logins, rejected logins) and any other logs I need to monitor.

For the Apache logs, I would also like the Rails logs. My sites are located in /srv/www/ and then the structure is sitename.com/public_html and sitename.com/logs. I have many sites on the server and I would like an easy way to view all of the errors and make some nice graphs out of them hence why I want to use Graylog2...

The log files in the logs folder are access.log and error.log.

The Rails logs would be in sitename.com/public_html/log. This contains production.log.

joschi
  • 20,747
  • 3
  • 46
  • 50
  • What is the question here? If you have rsyslog installed you can use it to send the Rails/Apache logs to Graylog2 (http://www.rsyslog.com/doc/imfile.html) – polynomial Sep 13 '11 at 06:12
  • have you try this? http://docs.graylog.org/en/1.2/pages/collector.html – zx1986 Nov 08 '15 at 12:59

4 Answers4

10

This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):

In Apache, I define a CustomLog format called graylog2_access which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.

Here is the custom format that it creates (human readable):

{ 
 "version": "1.1",
 "host": "%V",
 "short_message": "%r",
 "timestamp": %{%s}t,
 "level": 6,
 "_user_agent": "%{User-Agent}i",
 "_source_ip": "%a",
 "_duration_usec": %D,
 "_duration_sec": %T,
 "_request_size_byte": %O,
 "_http_status": %s,
 "_http_request_path": "%U",
 "_http_request": "%U%q",
 "_http_method": "%m",
 "_http_referer": "%{Referer}i"
}

For the Apache config, here is a copy/paste version:

LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access

Then in your host configuration:

CustomLog "|nc -u graylogserver 12201" graylog2_access
ETL
  • 6,443
  • 1
  • 26
  • 47
  • +1 Nice one! Have to try this on one of our systems, but with ERROR instead. – Henk Mar 22 '14 at 16:28
  • @Henk - if you do a format for Error log, please let me know, I want to but haven't taken the time yet. Also, check http://serverfault.com/questions/582510/apache-piping-log-to-netcat-fails for some info of things I ran into later on. – ETL Mar 22 '14 at 19:01
5

You can also send your log files to graylog2 server using this simple command:

tail -F -q $yourlogfile |   while read -r line ; do   echo "<7> $hostnamesendingthelog $line" | nc -w 1 -u $graylogserver 514;   done;

I use this mainly for test purposes to determine if my log format is adapted for easy querying in graylog2. For production use you will wan't to set up rsyslog or syslog-ng.

You can probably tail your rails log file and see what happens.

Pierig Le Saux
  • 51
  • 1
  • 1
  • I like the simplicity of this reply: Straightforward and no extra tools. I don't know what to do about the Log File timestamps: maybe nothing's to be done and they don't matter? – texas-bronius Oct 26 '17 at 14:06
2

Graylog2 only accepts logs in two formats: standard syslog and the Graylog extended log format (aka GELF). Arbitrary logs on disk are going to need some third-party process to consume the logs and translate it into a form that's useful for you.

Take a look at Logstash. Most people think of it as a tool to index logfiles using ElasticSearch, but it also contains a general purpose "log router" that lets you tail a bunch of files on disk and send them to a logging component like Graylog.

natacado
  • 3,317
  • 28
  • 27
  • 2
    Rsyslog can easily read the logs: http://www.rsyslog.com/doc/imfile.html – polynomial Sep 13 '11 at 06:11
  • 1
    While rsyslog can use other files as input, using logstash has the advantage of being able to use the predefined regular expressions of Grok (see http://logstash.net/docs/1.0.17/filters/grok) to get some structure into the log messages and not just plaintext. – joschi Dec 17 '11 at 19:08
  • syslog-ng can do the same with cleaner config compare to rsyslog: http://loggly.com/support/sending-data/logging-from/syslog/syslog-ng/#monitoring_a_file – HVNSweeting Mar 28 '13 at 03:28
2

You can use the apache2gelf scripts from here.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Anton
  • 29
  • 1