systemd terminates etcd service started with podman - reception only permitted for main PID

Question

I try to start etcd as a systemd service run in a container with podman.

After start I get this error log from systemd:

systemd[1]: etcd.service: Got notification message from PID 4696, but reception only permitted for main PID 4868

But etcd seem to be able to start an try to notify the container daemon:

21T15:31:08.817Z","caller":"etcdserver/server.go:2500","msg":"cluster version>
Aug 21 15:31:08 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:31:08.817Z","caller":"etcdmain/main.go:47","msg":"notifying init daemon>
Aug 21 15:31:08 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:31:08.818Z","caller":"etcdmain/main.go:53","msg":"successfully notified>

But systemd seem to not be aware of that and terminates the etcd service:

Aug 21 15:32:34 ip-10-0-0-71 systemd[1]: etcd.service: start operation timed out. Terminating.
Aug 21 15:32:35 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:32:35.000Z","caller":"osutil/interrupt_unix.go:64","msg":"received sign>
Aug 21 15:32:35 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:32:35.000Z","caller":"embed/etcd.go:367","msg":"closing etcd server","n>

This is the systemd service status:

$ sudo systemctl status etcd.service
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Sat 2021-08-21 15:32:35 UTC; 8min ago
    Process: 4868 ExecStart=/usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z --name etcd 842445240665.dkr.ecr.eu-nort>
   Main PID: 4868 (code=exited, status=0/SUCCESS)
        CPU: 3.729s

This is my systemd unit service file for etcd started with podman:

cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
After=podman_ecr_login.service mk_etcd_data_dir.service

[Service]
Type=notify
ExecStart=/usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z \
 --name etcd <my-aws-account>.dkr.ecr.eu-north-1.amazonaws.com/etcd:v3.5.0 \
 /usr/local/bin/etcd --data-dir=/etcd-data \
 --name etcd0 \
 --advertise-client-urls http://127.0.0.1:2379 \
 --listen-client-urls http://0.0.0.0:2379 \
 --initial-advertise-peer-urls http://127.0.0.1:2380 \
 --listen-peer-urls http://0.0.0.0:2380 \
 --initial-cluster etcd0=http://127.0.0.1:2380

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable etcd
sudo systemctl start etcd

I suspect that this might be related to Type=notify or perhaps the way I use podman or etcd. I start etcd in a similar way as described in etcd documentation: Run etcd clusters inside containers - Running a single node etcd. I run this on Debian 11 with Podman 3.0.1.

Any suggestions on how I can start etcd with podman as a systemd service?

Answer 1

According to this issue comment, these services should be run as Type=simple since they don't signal back to systemd. This PR to podman set it to Type=exec which also seem to work good.

After change to Type=exec in my service unit file, it now works:

$ sudo systemctl status etcd.service
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-08-21 15:59:23 UTC; 1min 28s ago
   Main PID: 4662 (podman)
      Tasks: 11 (limit: 442)
     Memory: 137.9M
        CPU: 3.576s
     CGroup: /system.slice/etcd.service
             ├─4662 /usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z --name etcd <my-aws-account>.dkr.ecr.eu-north-1.amaz>
             └─4846 /usr/bin/conmon --api-version 1 -c 616b317dc255ca86b308857dc6a180510fc166975a8a28437f3434111f03e7ad -u 616b317dc255ca86b308857dc6a>