nginx logging to access.log.1 instead of access.log, logrotate failing?

Question

I have an nginx instance that is set to log access to /var/log/nginx/access.log and errors to /var/log/nginx/errors.log, but as soon as logrotate runs each week, the file gets moves to *.log.1 and the new *.log file gets created, but nginx continues to log to the log.1 file instead of the new .log file (and nothing gets gzipped). The first time I noticed this, it had been 3 weeks since the log rotation and the log was getting huge. Running kill -HUP `cat /run/nginx.pid` made nginx start logging to the right place again, but the problem started again the next week.

The more important reason this is frustrating is that I have the logs set to upload to Loggly via rsyslog, and when nginx stops logging to the file I have rsyslog polling, then things stop uploading and I don't get any alerts.

I suspect it has something to do with restarting nginx, or reloading the config, because it didn't start until I had made a config change and reloaded the config in a way that I thought was normal. I tried running kill -USR1 `cat /run/nginx.pid` but the files continued to get logged to the wrong location until I ran kill -HUP `cat /run/nginx.pid`, which I already know does not solve the problem.

Any idea of what's going on? I admit I'm no expert on logrotate or nginx administration, but my Googles have failed me on this one.

Here is my nginx logrotate script, and let me know if there's anything else you might want to see. The nginx.conf has nothing special in it with regard to logging, other than defining the output locations.

/var/log/nginx/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
        endscript
}

EDIT: I think I found the problem. Here is the output of running the logrotate in debug mode:

$ sudo logrotate --force -d /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx

Handling 1 logs

rotating pattern: /var/log/nginx/*.log  forced from command line (52 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log needs rotating
considering log /var/log/nginx/error.log
  log needs rotating
rotating log /var/log/nginx/access.log, log->rotateCount is 52
dateext suffix '-20141023'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /var/log/nginx/access.log.1 does not exist
renaming /var/log/nginx/access.log.52.gz to /var/log/nginx/access.log.53.gz (rotatecount 52, logstart 1, i 52), 
renaming /var/log/nginx/access.log.51.gz to /var/log/nginx/access.log.52.gz (rotatecount 52, logstart 1, i 51), 
renaming /var/log/nginx/access.log.50.gz to /var/log/nginx/access.log.51.gz (rotatecount 52, logstart 1, i 50), 
renaming /var/log/nginx/access.log.49.gz to /var/log/nginx/access.log.50.gz (rotatecount 52, logstart 1, i 49), 
renaming /var/log/nginx/access.log.48.gz to /var/log/nginx/access.log.49.gz (rotatecount 52, logstart 1, i 48), 
renaming /var/log/nginx/access.log.47.gz to /var/log/nginx/access.log.48.gz (rotatecount 52, logstart 1, i 47), 
renaming /var/log/nginx/access.log.46.gz to /var/log/nginx/access.log.47.gz (rotatecount 52, logstart 1, i 46), 
renaming /var/log/nginx/access.log.45.gz to /var/log/nginx/access.log.46.gz (rotatecount 52, logstart 1, i 45), 
renaming /var/log/nginx/access.log.44.gz to /var/log/nginx/access.log.45.gz (rotatecount 52, logstart 1, i 44), 
renaming /var/log/nginx/access.log.43.gz to /var/log/nginx/access.log.44.gz (rotatecount 52, logstart 1, i 43), 
renaming /var/log/nginx/access.log.42.gz to /var/log/nginx/access.log.43.gz (rotatecount 52, logstart 1, i 42), 
renaming /var/log/nginx/access.log.41.gz to /var/log/nginx/access.log.42.gz (rotatecount 52, logstart 1, i 41), 
renaming /var/log/nginx/access.log.40.gz to /var/log/nginx/access.log.41.gz (rotatecount 52, logstart 1, i 40), 
renaming /var/log/nginx/access.log.39.gz to /var/log/nginx/access.log.40.gz (rotatecount 52, logstart 1, i 39), 
renaming /var/log/nginx/access.log.38.gz to /var/log/nginx/access.log.39.gz (rotatecount 52, logstart 1, i 38), 
renaming /var/log/nginx/access.log.37.gz to /var/log/nginx/access.log.38.gz (rotatecount 52, logstart 1, i 37), 
renaming /var/log/nginx/access.log.36.gz to /var/log/nginx/access.log.37.gz (rotatecount 52, logstart 1, i 36), 
renaming /var/log/nginx/access.log.35.gz to /var/log/nginx/access.log.36.gz (rotatecount 52, logstart 1, i 35), 
renaming /var/log/nginx/access.log.34.gz to /var/log/nginx/access.log.35.gz (rotatecount 52, logstart 1, i 34), 
renaming /var/log/nginx/access.log.33.gz to /var/log/nginx/access.log.34.gz (rotatecount 52, logstart 1, i 33), 
renaming /var/log/nginx/access.log.32.gz to /var/log/nginx/access.log.33.gz (rotatecount 52, logstart 1, i 32), 
renaming /var/log/nginx/access.log.31.gz to /var/log/nginx/access.log.32.gz (rotatecount 52, logstart 1, i 31), 
renaming /var/log/nginx/access.log.30.gz to /var/log/nginx/access.log.31.gz (rotatecount 52, logstart 1, i 30), 
renaming /var/log/nginx/access.log.29.gz to /var/log/nginx/access.log.30.gz (rotatecount 52, logstart 1, i 29), 
renaming /var/log/nginx/access.log.28.gz to /var/log/nginx/access.log.29.gz (rotatecount 52, logstart 1, i 28), 
renaming /var/log/nginx/access.log.27.gz to /var/log/nginx/access.log.28.gz (rotatecount 52, logstart 1, i 27), 
renaming /var/log/nginx/access.log.26.gz to /var/log/nginx/access.log.27.gz (rotatecount 52, logstart 1, i 26), 
renaming /var/log/nginx/access.log.25.gz to /var/log/nginx/access.log.26.gz (rotatecount 52, logstart 1, i 25), 
renaming /var/log/nginx/access.log.24.gz to /var/log/nginx/access.log.25.gz (rotatecount 52, logstart 1, i 24), 
renaming /var/log/nginx/access.log.23.gz to /var/log/nginx/access.log.24.gz (rotatecount 52, logstart 1, i 23), 
renaming /var/log/nginx/access.log.22.gz to /var/log/nginx/access.log.23.gz (rotatecount 52, logstart 1, i 22), 
renaming /var/log/nginx/access.log.21.gz to /var/log/nginx/access.log.22.gz (rotatecount 52, logstart 1, i 21), 
renaming /var/log/nginx/access.log.20.gz to /var/log/nginx/access.log.21.gz (rotatecount 52, logstart 1, i 20), 
renaming /var/log/nginx/access.log.19.gz to /var/log/nginx/access.log.20.gz (rotatecount 52, logstart 1, i 19), 
renaming /var/log/nginx/access.log.18.gz to /var/log/nginx/access.log.19.gz (rotatecount 52, logstart 1, i 18), 
renaming /var/log/nginx/access.log.17.gz to /var/log/nginx/access.log.18.gz (rotatecount 52, logstart 1, i 17), 
renaming /var/log/nginx/access.log.16.gz to /var/log/nginx/access.log.17.gz (rotatecount 52, logstart 1, i 16), 
renaming /var/log/nginx/access.log.15.gz to /var/log/nginx/access.log.16.gz (rotatecount 52, logstart 1, i 15), 
renaming /var/log/nginx/access.log.14.gz to /var/log/nginx/access.log.15.gz (rotatecount 52, logstart 1, i 14), 
renaming /var/log/nginx/access.log.13.gz to /var/log/nginx/access.log.14.gz (rotatecount 52, logstart 1, i 13), 
renaming /var/log/nginx/access.log.12.gz to /var/log/nginx/access.log.13.gz (rotatecount 52, logstart 1, i 12), 
renaming /var/log/nginx/access.log.11.gz to /var/log/nginx/access.log.12.gz (rotatecount 52, logstart 1, i 11), 
renaming /var/log/nginx/access.log.10.gz to /var/log/nginx/access.log.11.gz (rotatecount 52, logstart 1, i 10), 
renaming /var/log/nginx/access.log.9.gz to /var/log/nginx/access.log.10.gz (rotatecount 52, logstart 1, i 9), 
renaming /var/log/nginx/access.log.8.gz to /var/log/nginx/access.log.9.gz (rotatecount 52, logstart 1, i 8), 
renaming /var/log/nginx/access.log.7.gz to /var/log/nginx/access.log.8.gz (rotatecount 52, logstart 1, i 7), 
renaming /var/log/nginx/access.log.6.gz to /var/log/nginx/access.log.7.gz (rotatecount 52, logstart 1, i 6), 
renaming /var/log/nginx/access.log.5.gz to /var/log/nginx/access.log.6.gz (rotatecount 52, logstart 1, i 5), 
renaming /var/log/nginx/access.log.4.gz to /var/log/nginx/access.log.5.gz (rotatecount 52, logstart 1, i 4), 
renaming /var/log/nginx/access.log.3.gz to /var/log/nginx/access.log.4.gz (rotatecount 52, logstart 1, i 3), 
renaming /var/log/nginx/access.log.2.gz to /var/log/nginx/access.log.3.gz (rotatecount 52, logstart 1, i 2), 
renaming /var/log/nginx/access.log.1.gz to /var/log/nginx/access.log.2.gz (rotatecount 52, logstart 1, i 1), 
renaming /var/log/nginx/access.log.0.gz to /var/log/nginx/access.log.1.gz (rotatecount 52, logstart 1, i 0), 
rotating log /var/log/nginx/error.log, log->rotateCount is 52
dateext suffix '-20141023'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /var/log/nginx/error.log.1 does not exist
renaming /var/log/nginx/error.log.52.gz to /var/log/nginx/error.log.53.gz (rotatecount 52, logstart 1, i 52), 
renaming /var/log/nginx/error.log.51.gz to /var/log/nginx/error.log.52.gz (rotatecount 52, logstart 1, i 51), 
renaming /var/log/nginx/error.log.50.gz to /var/log/nginx/error.log.51.gz (rotatecount 52, logstart 1, i 50), 
renaming /var/log/nginx/error.log.49.gz to /var/log/nginx/error.log.50.gz (rotatecount 52, logstart 1, i 49), 
renaming /var/log/nginx/error.log.48.gz to /var/log/nginx/error.log.49.gz (rotatecount 52, logstart 1, i 48), 
renaming /var/log/nginx/error.log.47.gz to /var/log/nginx/error.log.48.gz (rotatecount 52, logstart 1, i 47), 
renaming /var/log/nginx/error.log.46.gz to /var/log/nginx/error.log.47.gz (rotatecount 52, logstart 1, i 46), 
renaming /var/log/nginx/error.log.45.gz to /var/log/nginx/error.log.46.gz (rotatecount 52, logstart 1, i 45), 
renaming /var/log/nginx/error.log.44.gz to /var/log/nginx/error.log.45.gz (rotatecount 52, logstart 1, i 44), 
renaming /var/log/nginx/error.log.43.gz to /var/log/nginx/error.log.44.gz (rotatecount 52, logstart 1, i 43), 
renaming /var/log/nginx/error.log.42.gz to /var/log/nginx/error.log.43.gz (rotatecount 52, logstart 1, i 42), 
renaming /var/log/nginx/error.log.41.gz to /var/log/nginx/error.log.42.gz (rotatecount 52, logstart 1, i 41), 
renaming /var/log/nginx/error.log.40.gz to /var/log/nginx/error.log.41.gz (rotatecount 52, logstart 1, i 40), 
renaming /var/log/nginx/error.log.39.gz to /var/log/nginx/error.log.40.gz (rotatecount 52, logstart 1, i 39), 
renaming /var/log/nginx/error.log.38.gz to /var/log/nginx/error.log.39.gz (rotatecount 52, logstart 1, i 38), 
renaming /var/log/nginx/error.log.37.gz to /var/log/nginx/error.log.38.gz (rotatecount 52, logstart 1, i 37), 
renaming /var/log/nginx/error.log.36.gz to /var/log/nginx/error.log.37.gz (rotatecount 52, logstart 1, i 36), 
renaming /var/log/nginx/error.log.35.gz to /var/log/nginx/error.log.36.gz (rotatecount 52, logstart 1, i 35), 
renaming /var/log/nginx/error.log.34.gz to /var/log/nginx/error.log.35.gz (rotatecount 52, logstart 1, i 34), 
renaming /var/log/nginx/error.log.33.gz to /var/log/nginx/error.log.34.gz (rotatecount 52, logstart 1, i 33), 
renaming /var/log/nginx/error.log.32.gz to /var/log/nginx/error.log.33.gz (rotatecount 52, logstart 1, i 32), 
renaming /var/log/nginx/error.log.31.gz to /var/log/nginx/error.log.32.gz (rotatecount 52, logstart 1, i 31), 
renaming /var/log/nginx/error.log.30.gz to /var/log/nginx/error.log.31.gz (rotatecount 52, logstart 1, i 30), 
renaming /var/log/nginx/error.log.29.gz to /var/log/nginx/error.log.30.gz (rotatecount 52, logstart 1, i 29), 
renaming /var/log/nginx/error.log.28.gz to /var/log/nginx/error.log.29.gz (rotatecount 52, logstart 1, i 28), 
renaming /var/log/nginx/error.log.27.gz to /var/log/nginx/error.log.28.gz (rotatecount 52, logstart 1, i 27), 
renaming /var/log/nginx/error.log.26.gz to /var/log/nginx/error.log.27.gz (rotatecount 52, logstart 1, i 26), 
renaming /var/log/nginx/error.log.25.gz to /var/log/nginx/error.log.26.gz (rotatecount 52, logstart 1, i 25), 
renaming /var/log/nginx/error.log.24.gz to /var/log/nginx/error.log.25.gz (rotatecount 52, logstart 1, i 24), 
renaming /var/log/nginx/error.log.23.gz to /var/log/nginx/error.log.24.gz (rotatecount 52, logstart 1, i 23), 
renaming /var/log/nginx/error.log.22.gz to /var/log/nginx/error.log.23.gz (rotatecount 52, logstart 1, i 22), 
renaming /var/log/nginx/error.log.21.gz to /var/log/nginx/error.log.22.gz (rotatecount 52, logstart 1, i 21), 
renaming /var/log/nginx/error.log.20.gz to /var/log/nginx/error.log.21.gz (rotatecount 52, logstart 1, i 20), 
renaming /var/log/nginx/error.log.19.gz to /var/log/nginx/error.log.20.gz (rotatecount 52, logstart 1, i 19), 
renaming /var/log/nginx/error.log.18.gz to /var/log/nginx/error.log.19.gz (rotatecount 52, logstart 1, i 18), 
renaming /var/log/nginx/error.log.17.gz to /var/log/nginx/error.log.18.gz (rotatecount 52, logstart 1, i 17), 
renaming /var/log/nginx/error.log.16.gz to /var/log/nginx/error.log.17.gz (rotatecount 52, logstart 1, i 16), 
renaming /var/log/nginx/error.log.15.gz to /var/log/nginx/error.log.16.gz (rotatecount 52, logstart 1, i 15), 
renaming /var/log/nginx/error.log.14.gz to /var/log/nginx/error.log.15.gz (rotatecount 52, logstart 1, i 14), 
renaming /var/log/nginx/error.log.13.gz to /var/log/nginx/error.log.14.gz (rotatecount 52, logstart 1, i 13), 
renaming /var/log/nginx/error.log.12.gz to /var/log/nginx/error.log.13.gz (rotatecount 52, logstart 1, i 12), 
renaming /var/log/nginx/error.log.11.gz to /var/log/nginx/error.log.12.gz (rotatecount 52, logstart 1, i 11), 
renaming /var/log/nginx/error.log.10.gz to /var/log/nginx/error.log.11.gz (rotatecount 52, logstart 1, i 10), 
renaming /var/log/nginx/error.log.9.gz to /var/log/nginx/error.log.10.gz (rotatecount 52, logstart 1, i 9), 
renaming /var/log/nginx/error.log.8.gz to /var/log/nginx/error.log.9.gz (rotatecount 52, logstart 1, i 8), 
renaming /var/log/nginx/error.log.7.gz to /var/log/nginx/error.log.8.gz (rotatecount 52, logstart 1, i 7), 
renaming /var/log/nginx/error.log.6.gz to /var/log/nginx/error.log.7.gz (rotatecount 52, logstart 1, i 6), 
renaming /var/log/nginx/error.log.5.gz to /var/log/nginx/error.log.6.gz (rotatecount 52, logstart 1, i 5), 
renaming /var/log/nginx/error.log.4.gz to /var/log/nginx/error.log.5.gz (rotatecount 52, logstart 1, i 4), 
renaming /var/log/nginx/error.log.3.gz to /var/log/nginx/error.log.4.gz (rotatecount 52, logstart 1, i 3), 
renaming /var/log/nginx/error.log.2.gz to /var/log/nginx/error.log.3.gz (rotatecount 52, logstart 1, i 2), 
renaming /var/log/nginx/error.log.1.gz to /var/log/nginx/error.log.2.gz (rotatecount 52, logstart 1, i 1), 
renaming /var/log/nginx/error.log.0.gz to /var/log/nginx/error.log.1.gz (rotatecount 52, logstart 1, i 0), 
running prerotate script
running script with arg /var/log/nginx/*.log : "
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
"
renaming /var/log/nginx/access.log to /var/log/nginx/access.log.1
creating new /var/log/nginx/access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/error.log to /var/log/nginx/error.log.1
creating new /var/log/nginx/error.log mode = 0640 uid = 33 gid = 4
running postrotate script
running script with arg /var/log/nginx/*.log : "
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
"
removing old log /var/log/nginx/access.log.53.gz
error: error opening /var/log/nginx/access.log.53.gz: No such file or directory

However, there are only archives up to about *.log.8.gz, so logrotate fails when it tries to interact with /var/log/nginx/access.log.53.gz. Why on earth is it trying to do that? I suppose I need to touch fake files to fill it out? This seems wrong somehow.

Answer 1

Bah, I finally found the answer after a long time digging. The problem, in my case wasn't that logrotate was failing. That error message is fine, and doesn't actually stop logrotate. The problem was that nginx was not releasing the file handle to the log file upon receiving the -USR1 signal from kill. Long story short, the reason it was not reloading the log files was because the /var/log/nginx folder was not owned by the same user as the nginx worker processes (owned by www-data, running under web). I have no idea how that changed (perhaps because this server was remade recently), but changing the folder to be owned by the same user as the nginx worker processes (and fixing the logrotate file to make new logs as web) fixed the issue.

Answer 2

I have the same problem in Ubuntu 14.04, but after reading your answer, I checked that my nginx is running under www-data, who owns the folder, so...

However, I have found this bug, which points on the mistake in the postrotate section in the /etc/logrotate.d/nginx.

To fix it, you could comment out

invoke-rc.d nginx rotate >/dev/null 2>&1

and the use any of the following options (from comments in that bug) instead:

• start-stop-daemon --stop --signal USR1 --quiet --pidfile /run/nginx.pid --name nginx
• nginx -s reload
• service nginx reload >/dev/null 2>&1

Answer 3

Another cause for this problem may be disk filling up.

NOTE: This may even happen with the access log filling up suddenly! DoS maybe?

To reproduce this you may have your access log fill up all of a sudden with many requests, once logrotate fires it will it will move access.log to access.log.1, but will fail to compress the next file, and therefore fail to send the USR1 signal to nginx.

How do you fix the problem?

• If you care about logs pipe the file over ssh + compress there, then bring it back. Be sure to analyze what caused access or error logs to fill up so quickly.
• If you don't care, truncate the file and reload nginx or delete and restart, be sure NOT to delete and reload as this will not release the inode (please don't say handle, it is a windows term)