15

Every time logrotate runs Apache/2.4.7 (Ubuntu) experiences a seg fault and does not restart:

[Wed Sep 10 06:35:54.266018 2014] [mpm_event:notice] [pid 20599:tid 140630283466624] AH00493: SIGUSR1 received.  Doing graceful restart
[Wed Sep 10 06:35:54.885118 2014] [core:notice] [pid 20599] AH00060: seg fault or similar nasty error detected in the parent process

My apache logrotate script looks like:

/var/log/apache2/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        /usr/sbin/apachectl graceful
    endscript
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi; \
    endscript
}

/srv/apache/log/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        /usr/sbin/apachectl graceful
    endscript
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi; \
    endscript
}

I keep the default logs in /var/log/apache2, but I keep vhost specific logs (for the three different vhosts hosted on this server) in the /srv/apache/log directory (e.g. mysite1_error.log, mysite1_access.log, mysite2_error.log, mysite2_access.log....).

The relevant parts of /etc/apache2/apache.conf:

#/etc/apache2/apache.conf
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel trace8   # I set to try and get more info about this problem.
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

where export APACHE_LOG_DIR=/var/log/apache2$SUFFIX. In my vhost conf files, such as /etc/apache2/sites-enabled/mysite1.conf I have:

#/etc/apache2/sites-enabled/mysite1.conf
LogLevel debug
ErrorLog  /srv/apache/log/mysite1_error.log
CustomLog /srv/apache/log/mysite2_access.log combined

and similar for the other sites.

Anyone know why logrotate is causing this crash?

One more thing:

Manually forcing the logrotate as root:

logrotate -v -f /etc/logrotate.d/apache2

does not cause the seg fault, but I know it's logrotate as I've tried playing with the times (weekly, daily) and the seg fault always occurs exactly as the logs are being rotated only.

How to reproduce on demand

# Set a crontab to run each minute (simulating cron.daily run of logrotate)
crontab -e
*/1 * * * * root /usr/sbin/logrotate -v -f /etc/logrotate.d/apache2 > /home/myuser/logrotate.log 2>&1

Finally

If I comment out the /srv/apache/log/*.log rotation block in /etc/logrotate.d/apache the seg fault does not occur either.

fpghost
  • 663
  • 1
  • 10
  • 22
  • Form the error message it looks like `/usr/sbin/apachectl graceful` is the offender (in theory scripts within /etc/logrotate.d/httpd-prerotate are also candidates). Can you confirm by running `/usr/sbin/apachectl graceful` manually? Is there any more hints when this happens in Apache's global errorlog (or the global syslog), maybe when you increase the LogLevel not only within but globally? – Nils Toedtmann Sep 10 '14 at 11:06
  • `/usr/sbin/apachectl graceful` manually at cmd line causes no problems (Also note that this line was originally `/etc/init.d/apache2 reload > /dev/null` but after getting the seg fault, I changed to `apachectl` on advice I read online, obviously it did not fix things). The LogLevel is already at the highest possible globally, `trace8` set in the `apache.conf` file. – fpghost Sep 10 '14 at 11:32
  • And just to rule it out: `/etc/logrotate.d/httpd-prerotate` does not exist. – fpghost Sep 10 '14 at 11:33
  • This is usually caused by a module not shutting down properly. Are you using any exotic modules? Anything added recently? 3rd-party modules? Try disabling them. – Giovanni Tirloni Sep 10 '14 at 12:18
  • So `/usr/sbin/apachectl graceful` is causing a segfault when issued by logrotate, but not when issued manually? Odd – Nils Toedtmann Sep 10 '14 at 12:41
  • @NilsToedtmann Even more specific in fact. A manual run of logrotate causes no isses (`logrotate -f -v /etc/logrotate.d/apache2`) but when ran by cron the segfault occurs. I'm going to update my OP with some further findings. – fpghost Sep 10 '14 at 13:05
  • @gtirloni `apachectl -M` shows my loaded modules as http://paste.ubuntu.com/8309365/ – fpghost Sep 10 '14 at 13:22
  • Can you try starting Apache through strace so you capture exactly where it's segfaulting? Stop it completely then run `strace -ff -o trace.log -r apachectl start` and attempt the graceful shutdown that will segfault it. – Giovanni Tirloni Sep 10 '14 at 13:24
  • It's easier to reproduce if you create a /etc/cron.d/logrotate-test with `*/1 * * * * root /usr/sbin/logrotate -v -f /etc/logrotate.d/apache2 > /home/myuser/logrotate.log 2>&1` – Nils Toedtmann Sep 10 '14 at 13:34
  • @NilsToedtmann Yes good point, I'll update my post. From my experiments so far it seems that if I just change the logrotate script to be a single block, i.e. `/var/log/apache2/*.log, /srv/apache/logs/*.log {....}` I don't get the segfault. So maybe it was just the two blocks were causing a second restart attempt during the first restart... – fpghost Sep 10 '14 at 13:58
  • Interesting. On my system `apache2ctl graceful & apache2ctl graceful & apache2ctl graceful & apache2ctl graceful &` doesn't segfault. That's 4 simultaneous graceful restarts. You get the same? – Ian Macintosh Sep 10 '14 at 14:57
  • (removed a stupid thought) – Nils Toedtmann Sep 10 '14 at 15:00
  • @NilsToedtmann If I enter `apache2ctl graceful & apache2ctl graceful & apache2ctl graceful & apache2ctl graceful &` my system DOES segfault! Strange, I wonder why mine does and yours does not. Oh well, I guess that is mystery solved....If anyone wants to add an answer to this effect I'd be glad to accept it. – fpghost Sep 10 '14 at 21:31
  • @fpghost you might want to add the answer yourself (that there seems to be a bad module preventing the reload, and basically forces you to stop/start apache. – Dennis Nolte Jun 15 '15 at 07:30

1 Answers1

14

It seems that if I just change the logrotate script to be a single block, i.e. /var/log/apache2/*.log, /srv/apache/logs/*.log {....} I don't get the segfault. So it was just the two blocks were causing a second restart attempt during the first restart...

If I enter apache2ctl graceful & apache2ctl graceful & apache2ctl graceful & apache2ctl graceful & on the cmd line my system DOES segfault, which seems to confirm

fpghost
  • 663
  • 1
  • 10
  • 22