2

I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS.

filebeat is already running in EKS to aggregate Kubernetes container logs.

I have configured rsyslog client with :

#/etc/rsyslog.d/50-default.conf
*.*;syslog;auth,authpriv.none   /var/log/syslog

# /etc/rsyslog.d/49-ship-syslog.conf
*.*;syslog;auth,authpriv.none action(
  type="omfwd"
  Target="logstash.dev.domain.com"
  Port="5000"
  Protocol="tcp"
)

and logstash with :

input {
  tcp {
    port => 5000
    type => syslog
  }
}

filter {
    grok {
        match => {
            "message" => "%{SYSLOGLINE}"
        }
    }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

I am able to do :

$nc -vz -u logstash.dev.domain.com 5000
found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
    outif (null)
    src 10.24.11.90 port 58291
    dst 47.176.158.158 port 5000
    rank info not available

Connection to logstash.dev.domain.com port 5000 [udp/commplex-main] succeeded!

also able to send test message :

echo -n "test message" | nc -4u -w1 logstash.dev.domain.com 5000

But don't see anything for tcpdump port 5000 in logstash OR in Kibana UI under Logstash shows Events Received 0

But when I login to logstash container and type echo -n "test message" | nc -4u -w1 logstash.dev.domain.com 5000 then I see this message showing up for tcpdump port 5000.

What I am missing here ?

Version : logstash:7.2.1 rsyslog 3.5

roy
  • 119
  • 1
  • 2
  • 12
  • Your ouput config is a bit off - remove the `document_type` as its a deprecated option, and `flush_size` is no longer an option (it may be the root cause of the issue). Even if the message is malformed, it'll be visible in Kibana with a `_grok_parse_error tag`. Can you confirm that the actual datagrams are making it to the host via `tcpdump udp port 514`? – Brennen Smith Sep 16 '19 at 23:36
  • 1
    rsyslog is sending using tcp (`Protocol="tcp"`) and logstash has an udp input configured (`input { udp { port => 5000`). (That can't work) – Marki Sep 19 '19 at 09:14
  • I have corrected the protocol, I tried both `UDP` & `TCP` – roy Sep 19 '19 at 13:05
  • setup a test nc and check if rsyslog sends anything to the port 5000 at all. Other than that: usual done? restarting/reloading, making sure the extra config files are actually included, have the correct permissions. did you start rsyslog directly instead of a daemon and checked if there is something missing? checked the manual? https://rsyslog.readthedocs.io/en/latest/configuration/modules/omfwd.html – Dennis Nolte Sep 19 '19 at 13:26
  • `rsyslog 3.5` <- are you sure? this is very-very-very old version of rsyslog! – Ektich Sep 22 '19 at 22:41

0 Answers0