0

We have rsyslog that save logs based on requirement to various directories:

/opt/logs/stats/dc1/stats_YYYY-MM-DD_%somename%.log
/opt/logs/stats/dc2/stats_YYYY-MM-DD_%somename%.log
/opt/logs/events/dc1/events_YYYY-MM-DD_%somename%.log
/opt/logs/events/dc2/events_YYYY-MM-DD_%somename%.log
/opt/logs/security/dc1/security_YYYY-MM-DD_%somename%.log
/opt/logs/security/dc2/security_YYYY-MM-DD_%somename%.log

How to archive these logs weekly ?

I would like them compressed and moved to a different folder:

/mnt/logs_archive/stats/dc1/stats_<YYYY-MM-DD>.tar.gz
/mnt/logs_archive/events/dc1/events_<YYYY-MM-DD>.tar.gz
/mnt/logs_archive/security/dc1/security_<YYYY-MM-DD>.tar.gz
/mnt/logs_archive/stats/dc2/stats_<YYYY-MM-DD>.tar.gz
/mnt/logs_archive/events/dc2/events_<YYYY-MM-DD>.tar.gz
/mnt/logs_archive/security/dc2/security_<YYYY-MM-DD>.tar.gz

Where YYYY-MM-DD is the date of the move that way we know that it shall include last 7 days of logs from the date.

Question:

  1. Is there a way to achieve this in a single/simple config
  2. Will I have to create a new one for log path?
  3. For Certain folders (security events) we are not interested in archiving logs for more than 120 days. can this be achieved too?

My current copy:

copytruncate
compress
dateformat _%Y-%m-%d.
dateext
extension log
olddir /mnt/archive_logs/

/opt/logs/stats/gw_stats*.log {
    rotate 48
    hourly
}

This gives me multiple compressed files

ll /mnt/archive_logs/


stats_YYYY-MM-DD_f2._2021-04-02.log.gz
stats_YYYY-MM-DD_asdas3._2021-04-02.log.gz

can I merge this into one stats_YYYY-MM-DD.gz ?

Also how do i use tar.gz ?

sarvesh.lad
  • 137
  • 5
  • show us please your current file of log rotate so we can see what you have already been done. you can use one config file to add all paths you need – djdomi Apr 02 '21 at 16:53

1 Answers1

1

You could write a single configuration file with different stanzas, one for each file to be rotated and use the directives dateext, dateformat, olddir, but be carefull with olddir (your /mnt dir suggests that the physical device is different so, probably you should make use of postrotate to move rotated files)

olddir directory

Logs are moved into directory for rotation. The directory must be on the same physical device as the log file being rotated, and is assumed to be relative to the directory holding the log file unless an absolute path name is specified. When this option is used all old versions of the log end up in directory. This option may be overridden by the noolddir option.

Your configuration file would look (assuming the names stats.log and events.log as the ones of log files).

compress
weekly
rotate 120
dateext
dateformat _%Y-%m-%d

/opt/logs/stats/dc1/stats.log {
   olddir /mnt/logs/archive/stats/dc1/
}

/opt/logs/events/dc1/events.log {
   olddir /mnt/logs/archive/events/dc1/
}

...
J.M. Robles
  • 865
  • 6
  • 9
  • Updating question with copy of what I have for stats there are multiple log files as stats_YYYY-MM-DD_.log I assume that I can just * to include everything – sarvesh.lad Apr 02 '21 at 18:10
  • Hmm, yes, but be carefull, (https://serverfault.com/questions/997734/log-rotation-not-working-properly/997772#997772 although using olddir that should not be a problem) and if the names have the date extension, you should not add it. – J.M. Robles Apr 02 '21 at 18:17
  • Thanks, so that gets me partially what I wanted, but I was hoping to archive all log files for a given week into one archive preferrbly tar split into 500 mb blocks each – sarvesh.lad Apr 02 '21 at 18:35
  • My guess is just move to old dir and and use post script to call tar however I want? – sarvesh.lad Apr 02 '21 at 18:37
  • With postrotate you adquire a better control. And the script does not seem to be too difficult – J.M. Robles Apr 02 '21 at 18:48
  • Thanks, but that compresses each file separately and I was hoping to make one big archive for all the logs inside the folder. But I got what I need and can work off it. – sarvesh.lad Apr 02 '21 at 19:08
  • Hi, One last. So I'm trying to get it work withtout copy truncate and it fails, you know why ? https://serverfault.com/questions/1059454/logrotate-with-nocreate-not-working – sarvesh.lad Apr 07 '21 at 15:04
  • Some programs use for logs files with an specific name to which they send the logs. If you move the log file, they send traces to a new empty file with that specific name. Some other programs, open a file that initially has a name and write to the descriptor of file. If the file is moved, they continue sending logs to the moved file .1, for example. In those cases, you can: 1) use copytruncate to avoid moving an make a copy instead of a move (and then empty the original file) 2) include in postrotate reset of services – J.M. Robles Apr 07 '21 at 16:45
  • thanks figured it out, used lastaction instead of postrotate to get what I needed – sarvesh.lad Apr 07 '21 at 16:48