1

We are getting syslog-ng feeds from a forwarder that we don't control.

The issue is that they are using an older version (RHEL 6), and while they have keep_hostnames set to 'yes' (chain_hostnames to 'no'), it's just not working. Their feed is still appending the relay's IP and timestamp to the message.

Other than create a global filter for this single input to rip off that leading IP and timestamp, I don't know any other way of doing it.

Anyone have any suggestions? I'm very surprised that I can't tell my syslog-ng to just ignore all appended relay chains.

jasonmclose
  • 161
  • 5
  • I'd settle for being able to filter on the sender's port. All messages are inbound at 514, but the sender is utilizing localport. If I could filter by the sender's port, I would be golden. Would be easy to tag messages after that. – jasonmclose Feb 07 '17 at 19:31
  • Hi, try to play with the keep-hostname and chain-hostname options on the syslog-ng server, the documentation has a table about how they should work in a relay scenario: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-options.html#global-options-chain-hostnames Also, check the values of the related macros on the server, like FULLHOST, – Robert Fekete Feb 08 '17 at 07:39
  • Thanks. I have it set correctly according to the table(s). The issue is really the upstream relay, as it keeps prepending its own IP and timestamp. I found a workaround, but it's not pretty. Will post below. – jasonmclose Feb 08 '17 at 13:25
  • Hi, on a second thought, you mention that the relay also appends a timestamp. Do you know why that happens? Is the relay configured to do so (using a template, for example), or is it possible that the message is not RFC-compliant, and syslog-ng cannot parse it properly (hence the IP and the timestamp)? – Robert Fekete Feb 08 '17 at 13:43
  • It's just prepending the extra source/hop like syslog-ng would do. spoof_source and keep_hostnames(yes) and chain_hostnames(no) should prevent this from happening, but I think that version has some bugs (older version of syslog-ng). – jasonmclose Feb 08 '17 at 17:28

1 Answers1

1

My work around for this isn't sexy, but it does work. I'm all ears if someone else knows of a better way.

Here is what I did. So in this situation, I'm trying to filter out IIS logs (s_net is simply tcp/udp listening on port 514). So here is what is sitting in my iis.conf file, inside of conf.d

parser p_iis_pattern_db {
    db_parser(file("/etc/syslog-ng/patterndb.d/strip_header.xml"));
};

filter f_iis {
    match("iis", value(".classifier.class"));
};

rewrite r_iis {
    set("${real_sender}", value("HOST"));
    set("iis", value("PROGRAM"));
};

destination d_iis {
    file("/nfs/syslog/iis/$YEAR/$MONTH/$DAY/${real_sender}/$YEAR-$MONTH-$DAY-$HOUR-${real_sender}-$PROGRAM.log"
        template("${orig_sender}${msg}\n"));
};

log {
    source(s_net);
    parser(p_iis_pattern_db);
    filter(f_iis);
    rewrite(r_iis);
    destination(d_iis);
    flags(final);
};

Then, I have a pattern file in patterndb.d (strip_header.xml), which looks like this:

<patterndb version='3' pub_date='2017-02-07'>
  <ruleset name='strip_sender' id='strip_leading_sender'>
    <rules>
      <rule id='iis' class='iis'>
        <patterns>
          <pattern>@ESTRING:datestamp: @@IPv4:real_sender@ iis - - - @ANYSTRING:msg@</pattern>
        </patterns>
      </rule>
    </rules>
  </ruleset>
</patterndb>

I would prefer to be filtering based upon a MACRO value other than the message, as I think this is probably rather expensive; it's having to introspect the message of every package. Because data is coming from a relay running an older version of syslog-ng, I perform an outbound tag of data on that relay, and the keep-hostname/chain-hostname options aren't working.

My other worry is that this is going to get sloppy in a hurry as I start adding in non-relay sources. I might have to do the whole thing where I name the conf files in a certain way to assure they are performed in the correct order, in order to save CPU cycles (is it alphabetical?).

jasonmclose
  • 161
  • 5