1

I have a number of CoreOS servers on Amazon AWS and would like to collect events or logs from them and forward them onto my ELK stack provider logz.io.

Being a little new to the ELK stack, I'm a little lost on how to best get the data out. It was suggested to me to use filebeat. But filebeat works with ordinary log files and most of the data is currently being fed into journald.

All my services are running on docker and I see that docker has the idea of logging drivers. Could I plug into that in some way?

There is a project called journalbeat which looks promising, but incomplete and currently forwards everything. I just want specific units and actually, specific events. So I may have to work on that project to make it do what I need. But, before I go further there, is there perhaps another way?

e.g. another lightweight service that I could run on the machine that might accept logging events and forward them on. So long as it's not logstash because that will consume too many resources on my already memory and cpu constrained aws instances.

I'm surely not the only person to run into this type of issue.

hookenz
  • 14,132
  • 22
  • 86
  • 142
  • You could always filter it, if you don't want everything. – Michael Hampton Apr 13 '16 at 21:48
  • I know I can filter it at the elasticsearch end, but then I don't want to be charged for uninteresting data. It costs to store it and on the account I currently have there is an amount of data per day. – hookenz Apr 13 '16 at 21:50

1 Answers1

1

There is a Docker log driver for 'gelf', and a input plugin for Logstash that understands gelf format. That is probably the most straightforward way.

Jason Martin
  • 4,865
  • 15
  • 24
  • Yes although it appears that logz.io doesn't accept gelf? I'll have to ask them. – hookenz Apr 13 '16 at 22:54
  • Also if gelf is udp I guess there is no end to end encryption, so that would be a no-go for me. – hookenz Apr 13 '16 at 22:57
  • Also I'm using docker 1.91 which doesn't yet support the -log-driver options. It looks like rsyslogd can be configured to pull from journald and that can then send to logz so I'm going to see if that might work. – hookenz Apr 14 '16 at 01:05