I search a way for a reliable export of journalctl logs.
I could use the --since=... option, but this is a bit fuzzy.
In my case a script would call journalctl --output=json every ten minutes.
I don't want to miss a single line and (if possible) I would like to avoid duplicate lines.
Some days after asking this question I came across RELP: https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol
You can install a syslog daemon such as rsyslog (the default on Red Hat derived systems). This will log all journal entries in a more backward compatible manner, and of course you can specify a custom log for whatever you wish.
If you don't need logs exported in real time, you can use journalctl --since as some people have mentioned. You can run it daily at midnight with the time specifier yesterday to get exactly 24 hours of logs.
If you really need to get logs at short intervals and don't want to miss a single entry, then you need to learn about the cursor. For each log entry journalctl will provide a cursor which can be used to skip to exactly that log entry with --cursor, or the immediately following log entry with --after-cursor. Consider the following sample JSON:
{
"__CURSOR" : "s=6ad7dcf190f3409c8bf8086fec22888c;i=286c44;b=6b134acc25e94d69b4713422b7c773be;m=46f7a97d25;t=55f5e93131a32;x=aecce3d8b96df5dc",
"__REALTIME_TIMESTAMP" : "1512234682620466",
"__MONOTONIC_TIMESTAMP" : "304802790693",
"_BOOT_ID" : "6b134acc25e94d69b4713422b7c773be",
"PRIORITY" : "6",
"_MACHINE_ID" : "770056613d554df2abcb7757ba2e6270",
"_HOSTNAME" : "dalaran.example.us",
"_PID" : "1",
"_UID" : "0",
"_SELINUX_CONTEXT" : "system_u:system_r:init_t:s0",
"_GID" : "0",
"_CAP_EFFECTIVE" : "3fffffffff",
"SYSLOG_FACILITY" : "3",
"SYSLOG_IDENTIFIER" : "systemd",
"_TRANSPORT" : "journal",
"_COMM" : "systemd",
"_EXE" : "/usr/lib/systemd/systemd",
"_CMDLINE" : "/usr/lib/systemd/systemd --switched-root --system --deserialize 25",
"_SYSTEMD_CGROUP" : "/init.scope",
"_SYSTEMD_UNIT" : "init.scope",
"_SYSTEMD_SLICE" : "-.slice",
"CODE_FILE" : "../src/core/unit.c",
"CODE_LINE" : "1505",
"CODE_FUNC" : "unit_status_log_starting_stopping_reloading",
"MESSAGE_ID" : "de5b426a63be47a7b6ac3eaac82e2f6f",
"MESSAGE" : "Stopping OpenSSH server daemon...",
"UNIT" : "sshd.service",
"_SOURCE_REALTIME_TIMESTAMP" : "1512234682615526"
}
For your purposes, the __CURSOR is an opaque blob. Just capture the value from the last log entry you receive on one call to journalctl and feed it to the next call:
journalctl --output=json --after-cursor="s=6ad7dcf190f3409c8bf8086fec22888c;i=286c44;b=6b134acc25e94d69b4713422b7c773be;m=46f7a97d25;t=55f5e93131a32;x=aecce3d8b96df5dc" ...
Use the --since option. To get logs from the last 10 minutes, just use:
--since -10m
That will give you logs 10 minutes prior to the current time. See the man page https://www.freedesktop.org/software/systemd/man/journalctl.html and this page on time specifications for systemd https://www.freedesktop.org/software/systemd/man/systemd.time.html#
one way of doing it (not very reliable, but can work):
$ crontab -l | tail -1
*/10 * * * * journalctl --output=json --since -10m >> journalctl.json
$
alternative way to do it; (assuming you need to import json output into elasticsearch):
syslog-ng can read from the journal and export to plain old text files. You can also setup syslog-ng to send data to other systems (including elasticsearch).
you can create python script for polling journalctl with using query_unique function. Probably running as a service could help also, with restart option -- to not to miss anything.
https://www.freedesktop.org/software/systemd/python-systemd/journal.html
