3

this morning I woke up to a 'service down' email I have received from watchdog. After connecting to the server I have found the docker service down. After little investigation, I have found that this morning Ubuntu made an unattended upgrade of the containerd service and after the upgrade, docker service was not started automatically.

How can I make sure that after the upgrade this will not happen again?

Krystian
  • 286
  • 1
  • 5
  • 18
  • This is a bug in the containerd package and [has already been reported](https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1870876?comments=all). – Gerald Schneider Dec 01 '20 at 07:47

3 Answers3

2

I set Docker and its related packages to hold so they are only upgraded on planned upgrades.

sudo apt-mark hold docker containerd

When I want to upgrade Docker I run apt-mark unhold, upgrade the packages, and hold them again. I have an Ansible playbook to automate this process. My playbook to install and configure Docker sets the packages to hold directly after the installation.

Gerald Schneider
  • 19,757
  • 8
  • 52
  • 79
  • Thanks for this. This acutally answers the question. The `containerd` upgrade fiasko was just an example of something that can happen. – Krystian Dec 01 '20 at 12:55
  • is your ansible playbook available somewhere, like ansible-galaxy, or whatever? – MaxiReglisse Dec 07 '20 at 15:49
  • No, it isn't, sorry. But it's pretty easy, you only need the [`dpkg_selections`](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/dpkg_selections_module.html) module for hold. – Gerald Schneider Dec 08 '20 at 06:59
1

We had the same issue. I opted for using a systemd unit file to create a loose dependency between containerd and dockerd.

Here's how:

As root:

Make the override directory (permission 755)

mkdir /etc/systemd/system/containerd.service.d/

Create the override file in the above directory (permissions 644)

vi override.conf

Add the following to the file

[Unit]
Before=docker.service
Wants=docker.service

Restart systemd daemon

systemctl daemon-reload

Check it works

Stop containerd

systemctl stop containerd

Check dockerd has stopped

systemctl status docker

Start containerd

systemctl start containerd

Check dockerd has started

systemctl status docker

++++++++++++++++++++++++++

Some links to other documentation on the subject

https://www.shellhacks.com/systemd-service-file-example/

https://man7.org/linux/man-pages/man5/systemd.service.5.html

Philip Lewis
  • 31
  • 4
0

The documentation on docker restart policies is helpful, start your containers with the always or unless-stopped options as appropriate for your use.

It also seems reasonable to stop Ubuntu from carrying out unattended upgrades.

user602730
  • 1