Context
I am experiencing something strange while integrating promtail to my stack
ERROR: for asmodius_promtail Cannot start service promtail: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/asmodius_nginx-logs/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged\\\" at \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx\\\" caused \\\"mkdir /var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx: read-only file system\\\"\"": unknown
The volume mentioned in this error is the one I use so promtail could read Nginx logs. (It only reads the files and does not need to write anything in the dir) Here is a portion of the docker-compose.yml
version: '3.7'
services:
nginx:
container_name: ${SERVER_NAME}_nginx
restart: always
networks:
- monitoring
- web
image: nginx:1.17.5-alpine
entrypoint: /entrypoint.sh
ports:
- 80:80
- 443:443
volumes:
- ./nginx/site.conf:/etc/nginx/conf.d/site.conf
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/entrypoint.sh:/entrypoint.sh
- nginx-logs:/var/log/nginx
- certificates:/etc/letsencrypt/
promtail:
restart: always
container_name: ${SERVER_NAME}_promtail
image: grafana/promtail:master-b652f0a
command: -config.file=/etc/promtail/docker-config.yaml
volumes:
- nginx-logs:/var/log/nginx
- /var/log:/var/log:ro
- ./promtail/docker-config.yaml:/etc/promtail/docker-config.yaml:ro
networks:
- monitoring
and the volume "inspection"
{
"CreatedAt": "2020-04-28T20:08:41Z",
"Driver": "local",
"Labels": {
"com.docker.compose.project": "asmodius",
"com.docker.compose.version": "1.24.1",
"com.docker.compose.volume": "nginx-logs"
},
"Mountpoint": "/var/lib/docker/volumes/asmodius_nginx-logs/_data",
"Name": "asmodius_nginx-logs",
"Options": null,
"Scope": "local"
}
Strangely enough, the very same docker-compose.yml
works like a charm on my Debian 9
- Docker version 19.03.8, build afacb8b7f0
- docker-compose version 1.22.0, build f46880fe
but fails on CentOS
- Docker version 19.03.8, build afacb8b
- docker-compose version 1.24.1, build 4667896b
Leads
I don't quite believe that docker-compose version could be the problem. I tried the following
- stop the entire stack, rm
nginx-logs
volume and start again - use
:ro
not use:ro
Questions
Since I do not entirely understand the ERROR
message, an explanation of the following elements might help me figure what I am doing wrong.
read-only file system\\\"\"": unknown
: I just don't get what"\"
is- why does it mention a
merge
? A merge with what? mkdir
.... so it tries to create a folder somewhere in/var/lib/docker
. But why does it have to do so for this specific volume while all the others (and there is quite a lot of them) succeed every time Idocker-compose up -d --force-recreate
(for it is how I add/update services on this machine)
All insights are welcome