How to send a signal to a Docker container without affecting restart policy?

Question

TIL that docker kill means kill in the sense of "make it dead", as opposed to the POSIX sense of "send a signal".

We have several containers that we need to send a SIGHUP to in order to reload configuration, but this causes them to ignore their restart policy of "always", which isn't what we want.

What's the best way to send signals to these containers without affecting their ability to auto-restart?

To more clearly demonstrate the issue we're seeing, take the following example.

We have some container that has a restart policy set to always

$ docker inspect cloudwatch-exporter | jq .[].HostConfig.RestartPolicy
{
  "Name": "always",
  "MaximumRetryCount": 0
}

We reload the config at some point using docker kill:

$ docker kill --signal=SIGHUP cloudwatch-exporter
cloudwatch-exporter

Some time later, something happens that kills the process. To simulate this, I'll send a signal within the container:

$ docker exec cloudwatch-exporter bash -c "kill 1"

At this point, the container is dead and won't restart:

$ docker ps -a | grep cloudwatch-exporter
c7827204bba5        prom/cloudwatch-exporter:cloudwatch_exporter-0.8.0                                "java -jar /cloudwat…"   20 hours ago        Exited (143) 3 minutes ago                            cloudwatch-exporter

What alternatives do we have for using docker kill? For many cases docker exec works, but it won't work for any container that only contains a single static-linked binary.

score 4 · Answer 1 · answered Sep 28 '20 at 08:37

4

This is a known Docker bug.

See https://github.com/moby/moby/issues/11065 "Non-fatal signals break restart policies"

See https://github.com/moby/moby/issues/41302 "Signal breaks unless-stopped restart policy"

Bypass the problem by doing docker exec "kill -HUP 1".

answered Sep 28 '20 at 08:37

FeepingCreature

141
3

1

....and 7 years later, they still didn't fix this behaviour :/ – Christopher Thomas Mar 02 '22 at 20:15

score 2 · Accepted Answer · answered Jul 24 '20 at 17:53

2

With the --signal option to docker kill?

docker kill --signal=SIGHUP <container>

Or with docker exec?

docker exec <container> killall -HUP nginx

answered Jul 24 '20 at 17:53

Michael Hampton

237,123
42
477
940

We're seeing docker kill interfere with the restart policy, which is what I'm trying to avoid. The kill and killall are usually shell builtins, so you'd have to wrap it in bash -c "killall -HUP nginx" or something. But that won't work with images that are single statically linked binaries. – leedm777 Jul 24 '20 at 18:57
Did you actually specify the signal? – Michael Hampton Jul 24 '20 at 19:01
Yes, we specified the signal. I edited the question to be more specific about what we're seeing. – leedm777 Jul 24 '20 at 19:06
You sent the signal correctly. Now you should investigate why the process is stopping. This focus on sending the signal seems like a red herring. – Michael Hampton Jul 24 '20 at 19:07
I'm not as concerned with why the process stopped as I am with why it didn't restart. FWIW, in the specific case I ran into today it was a startup timing issue with our deploy automation. We sent a SIGHUP to a container right after it started, and the container promptly died. My guess is that we sent the signal before it could register its signal handler. But that's irrelevant. Whether its a signal handler, segfault, or OOM, I'm expecting Docker to restart my container. The docker kill command stops that from working, so I'm looking for alternatives. – leedm777 Jul 24 '20 at 21:45

How to send a signal to a Docker container without affecting restart policy?

2 Answers2