Fluentd elasticsearch comprssion

Question

I forward logs from app servers with fluentd (type tail) to elasticsearch server. Do not understand the following:

Can I enable for reducing traffic the compression transmitted logs with type tail or another type?

Answer 1

If you just use a <match> of type elasticsearch that will send the data over via http calls.

One alternative would be to have 2 fluentd instances, one on the server where you tail and another one on one elasticsearch node and to communicate with one another via secure-forward in/out plugins.

These plugins allow transmission via TCP encrypted using TLS and that will also compress.

This way you take the load off the network and the data on the receiving fluentd instance will be pushed into elasticsearch fast since they are on the same machine.