Currently, we have the following setup:
Multiple AWS EC2 machines, some which have a Docker Container running, some are dockerless. To retrieve Syslogs, webserver logs (Apache) and application logs, we have a Fluentd agent (td-agent) running on EC2 instance. These are forwarding error log messages to a centralized Fluentd server (a log aggregator) which then in turn send them to Graylog. For access logs, all td-agents on the instances forward them directly to AWS Kinesis Firehose, which in turn stores those on S3 every 5 minutes (buffered) and will be searchable by AWS Athena. Access logs on Docker containers are written to stdout and error logs to stderr. The docker logging driver for fluentd is being used to forward them to the td-agent installed on the respective host machine.
Now, there are some problems with this setup:
- Access logs can't be viewed in real time (e.g. for debugging purposes by developers)
- The caching/buffering of the td-agents might become a problem on high loads
- The caching/buffering of the td-agents or Docker containers might become a problem when the log aggregator or the td-agents are out of service
We're not using CloudWatch Logs because of the price and other reasons. Working with real log files would also mean, that we'd need to rotate them regularly, pay attention to disk space etc. The last point might be tackled by using a RAM-Disk or a separate drive. But this would not resolve the actual problem of having a fixed size cache/buffer which might get full and blocking incoming logs.
What's a better approach on this problem? Are there any best practices regarding logging Docker containers?