4

We have a Java-application, that uses log4j and rotates its own logs daily:

  • The currently-used log-file is /var/log/foo/foo.log
  • It is moved into /var/log/foo/foo.log.YYYY-MM-dd when the day changes

We'd rather not change its configuration...

However, we do want to:

  1. Compress the file once it has been renamed
  2. Only retain a certain number of them

Though it wouldn't be difficult to write a cron-job to do both, we'd rather stay in the framework of logrotate...

I create the following /etc/logrotate.d/foo:

/var/log/foo/foo.log.* {
        daily
        rotate 2
        compress
        delaycompress
        missingok
        notifempty
}

but it does not do anything:

reading config file /etc/logrotate.d/foo

Handling 1 logs

rotating pattern: /var/log/foo/foo.log.*  after 1 days (2 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/foo/foo.log.2017-06-28
  log does not need rotating
considering log /var/log/foo/foo.log.2017-06-29
  log does not need rotating
considering log /var/log/foo/foo.log.2017-06-30
  log does not need rotating
considering log /var/log/foo/foo.log.2017-07-01
  log does not need rotating
considering log /var/log/foo/foo.log.2017-07-02
  log does not need rotating
considering log /var/log/foo/foo.log.2017-07-03
  log does not need rotating
considering log /var/log/foo/foo.log.2017-07-04
  log does not need rotating

How do I make it compress the files and delete the oldest ones, when their total number exceeds 2?

kasperd
  • 29,894
  • 16
  • 72
  • 122
Mikhail T.
  • 2,272
  • 1
  • 22
  • 49

3 Answers3

1

I really don't think you are going be able to do what you want with logrotate without some external commands. Also, the pattern /var/log/foo/foo.log.* would treat each individual file as if it was a separate log to be rotated, not as a set of rotated files.

man logrotate

Please use wildcards with caution. If you specify *, logrotate will rotate all files, including previously rotated ones.

Anyway, maybe something like this? Just set your rotate condition that will never be reached since log4j is already handling the rotation partly. Then put the rest in a prerotate script?

/var/log/foo/foo.log {
    size 100G # something big that will never be reached/rotated since log4j does this.
    missingok
    notifempty
    # delete the other logs over 7 days old
    prerotate
        find /var/log/foo/ -name 'foo.log.*' -mtime +7 -delete
        nice gzip /var/log/foo/foo.log.*
    endscript
}
Community
  • 1
Zoredache
  • 128,755
  • 40
  • 271
  • 413
  • Thanks. This just seems like a way to execute _arbitrary_ command -- there is nothing `logrotate`-ish about it :) I may as well just do a cron-job instead... – Mikhail T. Jul 05 '17 at 20:51
  • Yeah, like I said, I don't think you are going to find what you want. – Zoredache Jul 05 '17 at 21:52
  • Setting an unreachable rotate option doesn't work, `logrotate` script options (e.g. `prerotate`) are only run if at least one file is rotated (see `man logrotate`). I came up with a solution and posted it below. – Ryan Fisher Apr 03 '19 at 19:09
1

I needed to accomplish this behavior because the app teams wanted to maintain control of log and filename format as well as rotation schedule with log4j. If you read man logrotate you'll notice that all the pre/post/lastaction scripts only execute when at least one log was rotated. Thus, setting something unreachable like size 999G does not work.

Instead, what I did was set the copy and extension options so that:

  1. The original log file was never touched (it's managed by log4j)
  2. The logs rotated by logrotate have a different extension that I can filter on
/var/log/<service>/<service>.log {
    copy  # don't touch the orig. log file, it's managed by log4j
    extension .log  # End rotated files in ".log" so we can find and delete them
    rotate 1
    hourly
    missingok

    prerotate
        find /var/log/<service>/ -iname "*service.log.*" ! -iname "*.gz" -exec gzip {} \; && \
        aws s3 sync /var/log/<service>/ s3://<bucket>/<environment>/<ec2_id>/<service>/ --exclude "*" --include "*<service>*.gz" && \
        find /var/log/<service>/ -iname "*<service>.log.*" -mtime +7 -delete
    endscript

    lastaction
        find /var/log/<service>/ -iname "*<service>.*.log" -delete
    endscript
}

Yes, I could just make a cron job. My reasoning for using logrotate is that I keep all of the log rotation config management in a single Ansible role and in the future, I'll be making the case to disable log4j rotation and use logrotate instead which will be a simple change.

Ryan Fisher
  • 233
  • 1
  • 9
0

The Log4j Rolling File Appenders and the logrotate are not compatible. The log4j rotation will interfere with logrotate.

First you need to find the log4j configuration file for the application and replace the Rolling File Appenders with simple File Appenders. This will stop the log4j from rotating the files. Then you can use the logrotate to rotate the log files as you wanted.

If you can't modify the application's log4j configuration you can create your own one and specify this in the classpath when the application is stared.

-Dlog4j.configurationFile=path/to/log4j2.xml

Ref: https://logging.apache.org/log4j/2.0/faq.html#config_location

Oz.
  • 1