0

I've been in charge of a server that has been stressed beyond its tipping point for some time. I've recommended changes that are currently in the process of being implemented by other members of the team that should alleviate the core infrastructure problems of the site. But, there is an acute problem that requires a stopgap measure to buy us some time.

Apache restarts every morning at 7:30 due to the log rotation CRON running. 6:00 - 9:00 are our peak hours for normal site usage, and when combined with the CRON, the server runs out of memory.

What log rotation configuration would help? Should I move it from daily to hourly? Change the thresholds? Just change the times (I did this and was still met with an outage).

emsoff
  • 103
  • 4
  • Reconfigure apache to use a remote log server. Then it's not a problem at all. – EEAA May 08 '17 at 21:09
  • Am I missing something here, or would you just change the time logrotate starts in cron. That should take about 2 minutes. – Tim May 08 '17 at 22:27
  • @Tim as mentioned, I tried that but it still crashed. Albeit not as often. – emsoff May 08 '17 at 22:38

3 Answers3

2

EEAA's advice to use a remote log server addresses the problem completely, but may not be possible in your environment.

Use LogRotate to reload Apache rather than restarting it, as described on this website. You'll probably want to change the parameters to daily. It shouldn't make any difference to outage time if you rotate hourly, daily, or weekly.

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        /etc/init.d/apache2 reload > /dev/null
    endscript
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                run-parts /etc/logrotate.d/httpd-prerotate; \
        fi; \
    endscript
}

Reload should let Apache finish servicing any open requests before reloading the configuration. You may lose a few requests, but I wouldn't think you'd lose many. You may not lose any.

If you need higher reliability you should consider two load balanced servers with some kind of health check or live failover, or using a remote log server.

Tim
  • 30,383
  • 6
  • 47
  • 77
  • Thanks for the info, very helpful! I am going to let it run a week to see if it helps. I'll accept the answer as soon as I know! – emsoff May 11 '17 at 17:42
1

It's possible to rotate the logs without restarting or reloading apache at all. The downside is that you risk losing some log entries.

Here are two ways of doing that:

  • set up a separate log server and use whatever syslog software you prefer to forward the apache logs to that one. That way the log rotation is handled on the other server and won't affect apache. The downside is that you will increase the network traffic, and the additional CPU cycles and memory usage may tip the server over the edge since it's already close to it. Also, under heavy traffic some log lines may be lost in transit.
  • Instead of reloading apache in order to start writing to a new file, you can copy the contents of the file and then empty the old one. The upside is you don't restart apache. The downside is you're likely to lose a few log entries that might get written during the microseconds between the end of the copying and the zeroing of the file. For this option, the postrotate section of the config file should be changed to copytruncate.
Jenny D
  • 27,358
  • 21
  • 74
  • 110
  • Thanks for the info, very helpful! I've added some of these changes and am going to let it run a week to see if it helps. I'll accept the answer as soon as I know! – emsoff May 11 '17 at 17:41
-1

Change the cron scheduled time, even remove apache from logrotated and set it up manually if needed. Stop the apache service, pause for a few seconds to let apache really finish shutting down, rename the log file, restart apache. Since you only renamed the log file it stays in same directory and happens "instantly". Once your service is back up, then move/compress/whatever the logfile to where it needs to be.

If it gets really bad, my inner BOFH says to point the logs to /dev/null and don't worry about rotating them...

ivanivan
  • 1,448
  • 6
  • 6