I am deploying a docker-compose stack of 5 applications on a single AWS EC2 host with 32GB RAM. This includes: 2 x Java containers, 2 x Rails containers, 1 x Elasticsearch/Logstash/Kibana (ELK) container (from https://hub.docker.com/r/sebp/elk/).
When I bring the stack up for the first time, all containers start. The ELK container takes about 3 minutes to start. The others come up straight away.
But the ELK containers exits after about 5 minutes. I can see from the logs that the elasticsearch service will not start. The log messages indicates a memory limitation error.
However, when I then tear everything down, and bring it up again, all the containers start straight away, including the ELK container, and everything remains stable. The issue only occurs the first time I start the stack on a new EC2 instance.
I can see from the docker stats that the ELK container is only using 2-3GB of the 32GB RAM available on the instance.
The ELK container is configured as follows:
elk:
image: sebp/elk
hostname: elk
container_name: elk
volumes:
- ./pipeline/:/etc/logstash/conf.d/
tty: true
expose:
- "12201/udp"
network_mode: host
ports:
- "5601:5601"
- "9200:9200"
- "12201:12201"
ulimits:
nofile:
soft: 65536
hard: 65536
There are no dependencies between the containers on start up.
What is happening with elasticsearch when it first runs that cause the container to fail when starting?