syslog-ng prepending date to received logs, breaking patterns in patterndb

Question

I'm setting up central logging for our servers using syslog-ng + patterndb, however the logs the logging server is receiving from the client are prepended with the date, host and other data. This of course breaks all the patterns in patterndb so nothing matches.

Is there a way to do some preprocessing on the source log file before attempting to classify it or some other way to overcome this problem?

Cheers.

Relevant client conf:

source s_src {
   system();
   internal();
};

destination d_central_logging {
    syslog(192.168.1.1 transport("tcp") port("12345")); 
};

log {
    source(s_src);
    destination(d_central_logging);
};

Relevant server conf:

parser p_patterndb {
    db-parser(file("/var/lib/syslog-ng/patterndb.xml"));
};

source s_network {
    tcp(port(12345) flags(syslog-protocol));
};

filter f_class_unknown {
    match("unknown"
        value(".classifier.class")
        type("string")
    );
};

destination d_all {
    file("/tmp/all");
};

destination d_unknown {
    file("/tmp/unknown");
};

log {
    source(s_network);
    parser(p_patterndb);
    log {
        filter(f_class_unknown);
        destination(d_unknown);
    };
    log {
        destination(d_all);
    };
};

EDIT:

original log line: 10.0.2.2 - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

modified log line: Dec 23 13:59:08 192.168.33.44 264 <13>1 2014-12-23T13:42:50+00:00 devhost 10.0.2.2 - - [meta sequenceId="8"] - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

Shouldn't the source clause on the server use syslog() rather than tcp()? — Paul Haldane, Dec 24 '14 at 07:39
ah nuts - I was switching between different versions to see which worked best. Fwiw, I've settled on this one: `syslog(ip(192.168.1.1) port("12345")); ` - I'll re-run patterndb over the new output (which is less mangled but still prefixed) and update with what happens. — lsh, Dec 30 '14 at 19:38

score 0 · Answer 1 · answered Dec 29 '14 at 18:44

Paul is right, the protocols used in the client-destination and the server-source are mismatched. As a result, the server does not recognize the incoming message as a properly-formatted syslog message, and prepends the host, date and other information.

Regards,

Robert

score 0 · Accepted Answer · answered Jan 05 '15 at 13:24

ok, I've found the real reason why a match wasn't being made. I was using the update-patterndb command after confirming that the correct Apache patterns existed in the github.com/balabit/syslog-ng-patterndb repo. However, the update-patterndb command wasn't including .xml files, only .pdb files. I saw the compiled pattern file at /var/lib/syslog-ng/patterndb.xml changing but never checked it. As the man pages for update-patterndb say, it's only a thin wrapper around the pdbtool merge command. This works for me:

pdbtool merge -r --glob *.xml -D /etc/syslog-ng/patterndb.d/syslog-ng-patterndb/

The directory /etc/syslog-ng/patterndb.d/syslog-ng-patterndb/ is where I have the repo cloned. I can confirm the Apache patterns are now extracting the content as expected by inspecting the compiled pattern file, running the pdbtool test command and seeing the structured results in those that receive the data downstream.

syslog-ng prepending date to received logs, breaking patterns in patterndb

2 Answers2