rsyslog with logrotate: reload rsyslog vs copytruncate

Question

I'm working on Ubuntu 14 with the default rsyslog and logrotate utility.

In the default rsyslog logrotate /etc/logrotate.d/rsyslog config I see the following:

/var/log/syslog
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

From what I understand, it is recommended to use copytruncate in all logrotate scenarios, as it doesn't moves the current log, but rather truncates the log so any process with an open file handler will be able to keep writing to it.

So how come the default configuration using rsyslog reload feature instead?

score 37 · Accepted Answer · edited Mar 13 '18 at 13:00

To answer your question, you first need to understand the different trade-off of reload and copytruncate:

reload: the old log file is renamed and the process writing into that log is notified (via Unix signal) to re-create its log file. This is the fastest / lower overhead method: rename/move operations are very fast and have a constant execution time. Moreover, it is an almost atomic operation: this means that (nearly) no log entry will be lost during the move/reload. On the other hand, you need a process capable of reloading and re-opening of its log file. Rsyslog is such a process, so the default logrotate config use the reload method. Using this mode with rsyslog is strongly recommended by rsyslog upstream.
copytruncate: the old log file is copied into an archive file, and then it is truncated to "delete" old log lines. While the truncate operation is very fast, the copy can be quite long (depending of how big is your logfile). Moreover, some log entry can be lost during the time between the copy operation (remember, it can be slow) and the truncate. For these reasons, copytruncate is not used by default for services capable of reloading and recreate their log files. On the other hand, if a server is not capable of reload/recreate log files, copytruncate is your safest bet. In other words, it does not require any service-level support. The rsyslog upstream project strongly advises against using this mode.

I'm limiting my log files to 500M each, so copying them won't be a trouble(several seconds at most). Thanks! — Mattan, May 05 '15 at 07:57

score 28 · Answer 2 · answered Mar 13 '18 at 09:08

Speaking as rsyslog author, copytruncate is actually a very, very, very bad choice. It is inherently racy and using it is almost a guarantee that you will lose log data. The more frequently the file is written to, the more you will lose. And this is not just part of the last line, but can be several hundred ones, depending on the exact timing and state of the system at the time the rotation happens.

When the file is moved and a new inode (file) created, rsyslog keeps track of the previous file and complete processing. So you do not have any loss in this case. Guaranteed (except if you unmount the file system...).

On "reopenOnTruncate": I personally have seen reopenOnTruncate to be racy in other regards as well, especially with NFS and the like. Some time ago I totally removed that functionality, but got later persuaded into merging similar functionality back in. It'll stay "experimental" most probably forever, as I really know people run into trouble on very heavily loaded systems. "copytruncate" is simply no decent mode to work with log files.

I currently work on refactoring imfile (ETA 8.34 or 8.35). The refactored version will probably be able to prevent accidental re-send due to API race, but also cannot guard against data loss - because this is conceptually impossible.

score 2 · Answer 3 · answered May 05 '15 at 07:59

2

This depends completely on how the process is writing logs.
copytruncate only works, if the log messages are appended to the file (e.g. whatever >> logfile.
And not when it is redirecting the output (e.g. whatever > logfile).

answered May 05 '15 at 07:59

faker

17,326
2
60
69

score 1 · Answer 4 · answered Mar 02 '17 at 11:25

1

Since version 8.16 rsyslog has imfile option reopenOnTruncate that handles copytruncte issue.

answered Mar 02 '17 at 11:25

Selivanov Pavel

2,126
3
23
47

score 0 · Answer 5 · edited Apr 13 '17 at 12:14

For rsyslog specifically, it does probably make more sense to leave things as they are.

The basic reason is that rsyslog has internal queues it can use in cases where its output handle becomes unavailable.

The reload will a) cause rsyslog to recreate its own log file, and b) cause any queued events to be flushed to the file on creation.

It may be copytruncate does no harm (although I would be concerned about partially written lines being truncated), but I would tend to think that copy/delete/reload is 'safer' from an integrity point of view.

As mentioned by @faker, since rsyslog can handle the situation where its file becomes unavailable, there isn't a compelling reason to use copytruncate.

And as mentionned by @SelivanovPavel, rsyslog actually requires specific config to deal with copy truncate properly.

So if only because using the reload approach requires less deviation from default config, I would keep it.

rsyslog with logrotate: reload rsyslog vs copytruncate

5 Answers5