0

I'm trying to configure rsyslog to send logs to logstash on CentOS. So I was following the tutorial. However, after setting up, nothing happens. Everything started ok, not error occurred but no logs in elasticsearch.

Here is my /etc/rsyslog.conf:

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

*.*;\
 local3.none                                            -/var/log/syslog

*.*;\
local3.none                                             -/var/log/messages

*.* @@10.0.15.25:10514

And /etc/rsyslog.d/loghost.conf:

$ModLoad imfile
$InputFileName /var/log/devops_training.log
$InputFileTag devops
$InputFileStateFile stat-devops
$InputFileSeverity debug
$InputFileFacility local3
$InputRunFileMonitor

And here is my logstash config:

input {
  syslog {
    type => rsyslog
    port => 10514
  }
}

filter { }

output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "localhost:9200" ]
      index => 'rsyslog-%{+YYYY.MM.dd}'
      document_type => "rsyslog"
    }
  }
}

rsyslog config seems to have no errors:

rsyslogd: version 7.4.7, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

And the logs for logstash also doesn't have any errors:

[2017-06-07T20:11:48,004][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2017-06-07T20:11:48,188][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"adf934f1-caf5-48be-b65c-b2907c0d6336", :path=>"/var/lib/logstash/uuid"}
[2017-06-07T20:11:49,438][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-06-07T20:11:49,439][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-06-07T20:11:49,604][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x3fdb353a URL:http://localhost:9200/>}
[2017-06-07T20:11:49,623][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-06-07T20:11:49,744][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-06-07T20:11:49,758][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2017-06-07T20:11:49,880][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x8dcaeed URL://localhost:9200>]}
[2017-06-07T20:11:49,883][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-06-07T20:11:50,623][INFO ][logstash.pipeline        ] Pipeline main started
[2017-06-07T20:11:50,644][INFO ][logstash.inputs.syslog   ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2017-06-07T20:11:50,660][INFO ][logstash.inputs.syslog   ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2017-06-07T20:11:50,827][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

The problem is not only that I don't know hot to fix it. I cannot understand what is the problem and how to troubleshoot it.

Sergii Bishyr
  • 101
  • 4

2 Answers2

1

I suggest you split up the troubleshoting into two parts:

1.) Test if remote rsyslog forwarding works. Stop logstash and open a TCP connection with the following command:

nc -l 10514

On your client, log something to syslog with logger and see if it reaches your logstash server. You can also just restart rsyslog daemon to create some log traffic.

2.) Test if the connection between logstash and elasticsearch works properly. To do that define a simple file input in your logstash configuration and write some logs line into that file.

input {
    file {
        path => "/tmp/test_log"
        type => "rsyslog"
    }
}

Then check if your rsyslog indices are created correctly in elasticsearch.

yodave
  • 41
  • 2
1

I've setup something similar to this. So writing down a few steps I followed for troubleshooting. Check to see if the indices are created.

curl -XGET 'http://localhost:9200/rsyslog-*/_search?q=*&pretty'

Instead of starting logstash via systemctl, start it via CLI so you can see what's going on. The usual suggestion is to give a STDIN in input and STDOUT in output stanzas of logstash conf. However I just append the below line to output block.

stdout { codec => rubydebug }

and start logstash via commandline

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf

Then you'll be able to see the events as they're received and processed by logstash.

rav
  • 54
  • 5