Logrotate script not uploading to s3

Question

I am trying to use logrotate to upload log files from my rails and nginx commands once a day. When I run the logrotate command manually the files successfully upload, but it never does it on its own. I have tried using lastaction and also postrotate but neither actually uploads to S3. Is there something trivial that I'm missing?

/home/deploy/www/mysite.com/log/*log {
su
daily
dateext
dateformat _%Y_%m_%d
rotate 3
missingok
dateext
compress
delaycompress
notifempty
copytruncate
sharedscripts
lastaction
  rvmsudo passenger-config reopen-logs;
  /usr/bin/s3cmd sync /home/deploy/www/mysite.com/log/*.gz s3://mysite_com/logs/rails/
endscript
}

EDIT: Below is the corrected way of doing this. So instead of using the lastaction script, call a shell script instead.

lastaction
  ./rails_reload_and_upload.sh
endscript

which contains

#/bin/bash

./reload_rails_logs.sh
/usr/bin/s3cmd --config /home/username/.s3cfg sync /home/deploy/www/mysite.com/log/*.gz s3://mysite_com/logs/rails/

score 3 · Accepted Answer · answered Aug 04 '15 at 07:24

3

I'd say it's most likely that either rvmsudo or s3cmd need extra environment variables to run successfully, which aren't set in the context of logrotate, but which are in your interactive shell session. I wouldn't be surprised if rvmsudo isn't available in the restrictive PATH which cron sets.

answered Aug 04 '15 at 07:24

womble

95,029
29
173
228

2

Yup that was the issue. By emulating the cron environment using `env -i $SHELL --norc` I was able to figure out what configuration parameters I needed to use. – John Aug 04 '15 at 18:06

score 0 · Answer 2 · answered Mar 16 '22 at 07:19

If you are running EC2 servers, other than running a sperate bash script, one can add an Instance Profile to EC2 server so that aws s3 rsync does not need access key which cannot be assigned under cron job.

So the following logrotate config runs pretty well if Instance Profile added on the running EC2 server

/home/ubuntu//logs/*.log {
        copytruncate
        daily
        dateext
        rotate 30
        compress
        missingok
        su root root
        lastaction
                HOSTNAME=$(/bin/hostname -f)
                DD=$(/bin/date +%d)
                MM=$(/bin/date +%m)
                YYYY=$(/bin/date +%Y)
                BUCKET="s3://xxx-logs-archive/$YYYY/$MM/$DD/batch/$HOSTNAME/"
                FILE="/home/ubuntu/logs/batch.log-$YYYY$MM$DD.gz"
                /usr/local/bin/aws s3 cp "$FILE" "$BUCKET"
        endscript
}

Logrotate script not uploading to s3

2 Answers2