3

I have a docker container on ubuntu 16.04 and I want to run a docker command periodically. After checking the command runs successfully from the command line I've setup a crontab with the following:

*/1 * * * * sudo docker run {image-name} python3 /{path-to-scrip}/script.py

This seems to work ok until the script outputs something after which point the crons don't run and it stops docker from working with the following message:

docker: Error response from daemon: connection error: desc = "transport: dial unix /var/run/docker/containerd/docke
r-containerd.sock: connect: connection refused".

I've read that stdout may be the problem and I've tried the following in crontab:

*/1 * * * * sudo docker run {image-name} python3 /{path-to-scrip}/script.py > /home/logs 2>&1

This didn't fix the problem and the logs file remains empty.

This is running on a virtual machine and restarting the instance gets me back to square one where docker runs.

I'm not the most experienced sys admin and suspect I've made a naive mistake, but I don't know how to progress this? What should I have done to setup the cron?

Additional info

I'm not sure if it's relevant, but this is running on a google cloud engine instance and the firewall is set to prohibit all external connections.

Update

After a hunch I've got it running on CentOS7, the cron job worked for a number of hours, but then the script gave an error. The error is something to be expected from the script, but this has lead to the same docker message and cron no longer works.

Further update From this thread I decided to run docker inspect --format '{{json .State }}' {container id for failed run} - this shows the following:

    {"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":8204,"ExitCode":0,"Error":"","StartedAt":"2018-01-02T03:1
5:03.016836367Z","FinishedAt":"0001-01-01T00:00:00Z"}

I guess I just need to figure out how to stop the process from running in order to shut the container down.

goose
  • 175
  • 1
  • 7
  • So, what do you get in `/home/logs`? – tripleee Jan 01 '18 at 13:06
  • Hi @tripleee - /home/logs is empty – goose Jan 01 '18 at 13:32
  • The error message looks like the Docker service dies. But as a first step, maybe use `docker run --rm -it` so you can actually capture its output. (The `--rm` is unrelated, but probably a very good idea. You may find that `dcker ps -a` reveals a great number of hosed copies for you to clean up.) – tripleee Jan 01 '18 at 13:53
  • 1
    Adding your user to the `docker` group so you don't need `sudo` to run simple Docker commands will probably also be a very good idea. – tripleee Jan 01 '18 at 13:55
  • @tripleee turns out I hadn't picked a directory I could write to, the logs now show "the input device is not a TTY". From advice from this thread (https://stackoverflow.com/questions/43099116/the-input-device-is-not-a-tty) I tried --rm -i, but the problem has now come back whereas with the previous settings it seemed to run the command at least once, and although it stopped the connection refused message didn't come back – goose Jan 01 '18 at 16:48
  • @tripleee - the logs also now show the same message: docker: Error response from daemon: connection error: desc = "transport: dial unix /var/run/docker/containerd/dock$ time="2018-01-01T16:58:04Z" level=error msg="error waiting for container: context canceled" – goose Jan 01 '18 at 16:59
  • Vaguely related? https://github.com/moby/moby/issues/34284 – tripleee Jan 01 '18 at 18:39
  • @Tripleee thanks for that, the docker dev thinks it's an is problem, maybe I should try a different distribution of Linux then. Only really used ubuntu, but guessing I can crack this on one of the similar distributions. – goose Jan 01 '18 at 19:21
  • @tripleee - I got this working on CentOS for a while, but eventually the script hit an error and brought back the same docker connection issue. Since it worked until the error, it now seems to be related to the script not terminating in a well behaved manner. I just need to figure out how this upsets docker. – goose Jan 02 '18 at 10:17

1 Answers1

1

I'm not putting this here as the best possible answer, if I could mark this as the correct answer I clearly wouldn't, however for anyone coming across something similar I found that the docker image wasn't very fault tolerant, even when I tried on CentOS 7. The first thing I needed to do was to liberally include try/catch blocks in my script.

Secondly, to unblock docker I created the following bash script:

#!/bin/bash
sudo service docker stop
sudo service docker start
sudo docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

This enabled docker to continue to run, I have no idea why this combination of commands was needed in my case. I may add a cron command for this script if I see the same error again in the future.

goose
  • 175
  • 1
  • 7