Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [fluentd]

Fluentd is an open source log aggregator.

Fluentd is a fully free and fully open-source log collector that instantly enables you to have a "Log Everything" architecture with 125+ types of systems.

Fluentd treats logs as JSON, a popular machine-readable format. It is written primarily in C with a thin-Ruby wrapper that gives users flexibility.

Fluentd's performance has been proven in the field: its largest user currently collects logs from 5000+ servers, 5 TB of daily data, handling 50,000 msgs/sec at peak time.

35 questions
5
votes
0 answers

Consuming multiline json in fluentd

I am new to fluentd. I have applications that run in Docker containers. They are Java apps that log in JSON format. The JSON messages are usually split over multiple lines. I would like to use the Docker fluentd log driver to send these messages to…
wimnat
  • 299
  • 2
  • 10
5
votes
1 answer

How to maximize utilization of fluentd server?

I have a fluentd server that is processing an average of 1000 req/s. This server is composed by 32 cpus. In this server are running 32 fluentd docker containers with the same configuration. The entry point of the server is an nginx as a load…
MiquelB
  • 53
  • 4
4
votes
1 answer

logstash-forward equivalent for fluentd?

Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd? I am trying to send log files from an application to a remote fluentd but have not seen whether this is possible via an agent somehow. I understand that you can…
adamo
  • 6,867
  • 3
  • 29
  • 58
3
votes
3 answers

Disable fluentd on on Container-Optimized OS (GCE)

I would like to disable fluentd on a VM instance running Container-Optimized OS on Google compute engine. The reason being that it is eating up a lot of RAM for functionalities I do not require. $ ps aux --sort -rss USER PID %CPU %MEM VSZ…
Nils André
  • 131
  • 5
3
votes
1 answer

How to specify the log tag from docker compose on google logging?

In my docker compose file, I can specify the logging driver as a "fluentd" and by default, it pushes the log to the listener of google-fluentd on port 24224 I can see the log appear on the logging page along with the tag as the container Id.…
Thai Tran
  • 183
  • 11
3
votes
2 answers

fluent configuration not working

I have setup fluent with elasticsearch and Kibana with nginx. I followed the instruction and when I go to http:/192.168.10.25:8888 I get the following message 400 Bad Request 'json' or 'msgpack' parameter is required Below is my fluent.conf ##…
rahrahruby
  • 577
  • 5
  • 12
  • 20
2
votes
0 answers

Monitoring Cassandra Server Logs?

I've done some searching and have not found any examples of people using something like FluentD to forward Cassandra log file data (system.log) to a log indexer. Is this something any is doing? If not, why? If yes, are there example fluentD configs…
Drew
  • 123
  • 4
2
votes
0 answers

fluentd modular configuration

Is there any way to setup fluentd/td-agent in a way that it's configuration will be modular? I know there is @include directive but this works only if every time I add something new I modify main td-agent.conf file adding new rule to replace tag…
termil0r
  • 123
  • 1
  • 6
2
votes
2 answers

How to collect logs for statistics in site?

I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing. I see 2 variants: write logs to local log files for each servers rename logs for template…
Bdfy
  • 181
  • 1
  • 1
  • 3
2
votes
1 answer

Forward linux logs to fluentd on linux

On one VM I have this in /etc/rsyslog.d/50-default.conf *.* @192.168.29.1:42185 # Default rules for rsyslog. On the vm with fluentd I have this: I have this in /etc/td-agent/td-agent.conf type syslog port 42185 tag …
James Black
  • 133
  • 1
  • 8
2
votes
1 answer

GCE stackdriver logging agent (fluentd) memory leak with COS

I have a VM on GCE where I run a custom Docker image. I installed it on COS (cos-stable-74-11895-125-0) to a g1-small (1 vCPU, 1.7 GB memory) instance. It is an Elasticsearch with locked memory settings. It consumes exactly 1 GB of RAM. The setup…
Krisztián Velez
  • 121
  • 2
1
vote
0 answers

Is there a preferred way to migrate from logcheck to fluentd?

Logcheck uses a negative filter approach, you specify the log messages that you consider harmless and you will see the rest. Looking at fluentd it is the opposite, it will filter messages that are deemed interesting, and show(forward) you…
Keve
  • 11
  • 1
1
vote
0 answers

Routing Fluent Bit tcp event doesn't match to Fluent

i’m new to Fluent and Fluent Bit. Fluent Bit (0.13) listens to tcp and forwards to a Fluent instance (td-agent 3.2). Fluent will push it to rabbitmq. The basic process is working. Fluent Bit generates messages like 2018-07-09T11:44:48+02:00 …
SineMatha
  • 11
  • 1
1
vote
1 answer

How to retrieve logs from AWS EC2 and Docker containers fail-proof?

Currently, we have the following setup: Multiple AWS EC2 machines, some which have a Docker Container running, some are dockerless. To retrieve Syslogs, webserver logs (Apache) and application logs, we have a Fluentd agent (td-agent) running on EC2…
Roger Lehmann
  • 131
  • 1
  • 7
1
vote
0 answers

Microsoft OMSAgent Listens on TCP port 16001

I have installed the Microsoft OMSAgent (on Linux). Now I see that it listens on TCP port 16001. Neither the documentation nor grepping through the Sourcecode gave some hints why. Anyone got an idea what services usually uses port 16001? fluentd?…
arved
  • 453
  • 2
  • 14
1
2 3