5

I'm trying to centralise logging in an environment that using multiple application technologies (Java, Rails and various DBs).

We want to developers to bring up stacks with Docker Compose, but we want to them to refer to a central log source (ELK) to debug issues, rather than trying to open shells into running Docker containers.

The applications all write to the file system rather than to STDOUT/STDERR, which removes all of the options associated with the Docker logging driver, and logspout too.

What we have done is configure the containers to have rsyslog include the application log files and forward those to logstash which has a syslog input. This works in terms of moving the logs from A to B, but managing multi-technology logs in ELK based on the syslog input is horrible (eg trying to capture multine Java stacktraces, or MySQL slow queries).

Is there a better way to do this? Should I be running logstash in each container, so that I can apply filters and codecs directly to the log files, so that I don't have to rely on the syslog input?

Of is there some way to use the Docker logging driver with application log files that are written to the file system?

Garreth McDaid
  • 3,399
  • 26
  • 41
  • I was able to make some progress on this be designing the Docker image to tail the relevant logs at ENTRYPOINT. eg tail -f /var/log/tomcat7/*. When you do that, the output of the tail is part of STDOUT, so hopefully now I can use some of the Docker log driver options. – Garreth McDaid Mar 21 '17 at 16:43
  • Confirmed that if you tail -f the logs at ENTRYPOINT you can use the Docker log drivers options. Note, for docker-compose, where logstash is a service, extra config is required. – Garreth McDaid Mar 24 '17 at 11:26

2 Answers2

2

Recent versions of Docker support transmitting logs in 'GELF' format to a network port. Logstash has a GELF input. You could run Logstash on every node and have all Docker instances on the node forward to it.

As a Logstash input: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html

gelf {
}

For Docker output: https://docs.docker.com/engine/admin/logging/overview/#gelf

$ docker run -dit \
             --log-driver=gelf \
             --log-opt gelf-address=udp://127.0.0.1:12201 \
             alpine sh

(The gelf-address is from outside the containers perspective, not inside)

Jason Martin
  • 4,865
  • 15
  • 24
  • But as I understand it, the Docker logging driver only handles application logs that write to STDOUT/STDERR. – Garreth McDaid Mar 21 '17 at 15:42
  • @GarrethMcDaid Oh, I see what you mean. Running logstash internal to the container is quite a bit of overhead. You'd need them to all emit to STDOUT. – Jason Martin Mar 21 '17 at 15:52
  • This is pretty much the correct answer, but with caveats. In your ENTRYPOINT, you have to tail -f your applications logs, so they go to STDOUT. You can then pipe your logs to a logstash GELF input. – Garreth McDaid Mar 24 '17 at 11:25
0

You could also configure logstash to parse the various json log files docker produces by default.

Another approach is to use what is called a sidecar in Kubernetes.

They provide a few different examples in their cluster logging concepts page.

How you choose to apply that concept is entirely dependent on your needs.

However, a simple proof of concept might work by:

  • creating a logstash sidecar that accepts incoming syslog streams (e.g uses the syslog input)
  • configuring all containers to use docker's syslog driver to send logging to the sidecar.

You could also of course set up a central syslog listener (using logstash, or rsyslog for example), and do this without a sidecar.

This approach is also very much in the same vein as @Jason Martin's suggestion to use GELF.

Another use of a local sidecar might be to create a container running logstash with a file input, and exposed a log volume (e.g. /var/log/, or /logs). You could then share that volume with other containers, to allow them to write their logs (e.g. /logs/$INSTANCE_ID/file.log), and have logstash parse them.

This last setup allows monitoring files rather than STDOUT/STDERR, but you will probably have to have your log directory chmod 1777 (or have several such sidecars).

The 'reverse' setup would also work, of course (but seems harder to manage/maintain): have your application containers expose a log volume, and have a logstash sidecar deal read the log volume's content.

iwaseatenbyagrue
  • 3,588
  • 12
  • 22
  • Again, there is a dependency here for the application logs to write to STDOUT, which they don't by default. – Garreth McDaid Mar 21 '17 at 16:42
  • @[GarrethMcDaid](https://serverfault.com/users/179901/garreth-mcdaid), the last suggestion does not - the last one uses volume sharing to allow monitoring _actual_ logs files, rather than stdout/stderr. It just edited it to highlight as much (since I agree I didn't really make that clear). – iwaseatenbyagrue Mar 21 '17 at 16:45