Podman rootless journald logging

Question

I'm trying to log to the host's systemd-journald from a rootless podman-container. When i run the container as root with the --privileged flag, i can read the logs from the container on the host with journalctl as expected. However, running the container in rootless mode breaks said logging-functionality (nothing shows up in jornalctl). Is there any way to solve this?

Answer 1

In case your system looks like this

$ grep Storage /etc/systemd/journald.conf 
#Storage=auto
$ ls /var/log/journal
ls: cannot access '/var/log/journal': No such file or directory
$ 

(That seems to be the default for CentOS 8.3)

you could try

sudo mkdir /var/log/journal

and then reboot the computer.

(That solution worked for me on CentOS 8.3)

Explanation

If Storage is set to auto in the file /etc/systemd/journald.conf and the directory /var/log/journal is missing, the journal log data will only be stored in memory.

(auto is also the default in case Storage has not been set in the file /etc/systemd/journald.conf)

See also man journald.conf where this mode is called volatile.

There is a GitHub issue in the Systemd repository with the title Allow users to read their own volatile journals. I would guess that that sudo mkdir /var/log/journal will no longer be needed after that GitHub issue has been fixed.

Answer 2

As seen on this github comment, try

--log-driver journald for podman create/run. With that option conmon will write the container output to journald.

The journald log driver has been available in podman since at least release 1.6.4 (2019)

Do not use the journald event logger when it is not available

I've tested this over the last few days with great success. Some of my containers are able to write syslog after bind mounting /dev/log inside the container. However, recent CUPS servers with explicit journald support don't get picked up writing to /dev/log. Altering this daemon to write to stderr and then running the container with --log-driver journald allowed logging to be picked up by journalctl and actually stored to /var/log/messages as well.

My environment is RHEL8, podman 4.0.1 and podman-compose 0.1.7. I see from that github issue that more recent versions of podman-compose support docker-compose.yml directives for logging.

Since I came across this question as the top Google search for my problem, I hope recording this answer to my problem and yours will help the next poor soul searching for ways to make rootless podman write to journald!