One question about logrotation on a dedicated webserver (64gb RAM) with apache and mysql. In high load hours (2k users online every moment) we are experiencing a strange behaviour.
As an example, at 16:30 we have 1850 users, low ram utilisation (20gb), no slow queries logged and everything is working fine.
At 16:50, logs rotate.
At 16:53, the server begins crashing and cannot handle anymore the users. Page load becomes incredibly slow, and goes worse until we manually restart apache.
As we are under apc and as far as i know logrotation causes apache to reload, may the logrotation be the cause of degradation in performance? Maybe because of clearing apc's cache?
Here the logrotate configuration:
/var/log/apache2/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
/etc/init.d/apache2 reload > /dev/null
endscript
}
and the top I/O using processes
----total-cpu-usage---- -most-expensive- -dsk/total- ----most-expensive----
usr sys idl wai hiq siq| cpu process | read writ| block i/o process
3 0 97 0 0 0|mysqld 0.6|1984B 430k|mysqld 16B 297k
3 0 96 0 0 0|mysqld 0.8| 0 732k|mysqld 0 584k
6 1 94 0 0 0|mysqld 0.6| 0 104k|mysqld 0 312k
4 1 96 0 0 0|apache2 0.6| 0 196k|rotatelogs 0 4096B
4 1 94 0 0 0|apache2 1.5| 0 452k|jbd2/sda2-8 0 284k
3 1 96 0 0 0|apache2 1.4| 0 240k|jbd2/sda2-8 0 104k
5 1 95 0 0 0|mysqld 1.1| 0 0 |apache2 0 4096B
6 1 93 0 0 0|apache2 1.2| 0 252k|jbd2/sda2-8 0 96k
3 1 96 0 0 0|apache2 0.5| 0 4260k|apache2 0 1768k
7 1 93 0 0 0|apache2 1.8| 0 436k|apache2 0 648k
5 1 94 0 0 0|mysqld 1.1| 0 12k|apache2 0 12k
2 0 98 0 0 0|apache2 0.2| 0 1180k|mysqld 0 1032k
6 1 92 0 0 0|apache2 1.5| 0 480k|mysqld 0 664k
4 0 96 0 0 0|apache2 0.5| 0 312k|rotatelogs 0 8192B
5 0 95 0 0 0|mysqld 0.9| 0 0 |apache2 0 4096B
3 1 95 0 0 0|apache2 0.6| 0 516k|mysqld 0 308k
4 1 95 0 0 0|apache2 0.9| 0 316k|mysqld 0 296k
6 0 93 0 0 0|apache2 2.1| 0 332k|jbd2/sda2-8 0 176k
5 1 94 0 0 0|apache2 1.6| 0 420k|apache2 0 40k
3 0 96 0 0 0|mysqld 0.8| 0 0 |mysqld 0 412k
5 0 94 0 0 0|apache2 2.1| 0 188k|jbd2/sda2-8 0 52k
8 1 91 0 0 0|apache2 2.1| 0 544k|apache2 0 232k
7 0 93 0 0 0|mysqld 1.2| 0 2020k|mysqld 0 2228k
3 0 96 0 0 0|mysqld 0.6| 0 212k|rotatelogs 0 4096B
3 0 97 0 0 0|apache2 0.6| 0 0 |apache2 0 8192B
6 0 94 0 0 0|apache2 1.8| 0 320k|jbd2/sda2-8 0 120k
7 1 92 0 0 0|mysqld 1.6| 0 656k|mysqld 0 832k
6 0 93 0 0 0|mysqld 1.2| 0 312k|jbd2/sda2-8 0 80k
6 1 94 0 0 0|apache2 1.2| 0 548k|jbd2/sda2-8 0 140k
5 0 95 0 0 0|apache2 1.5| 0 128k|jbd2/sda2-8 0 20k
4 0 95 0 0 0|apache2 1.6| 0 152k|jbd2/sda2-8 0 36k
2 0 98 0 0 0|mysqld 0.4| 0 12k|apache2 0 4096B
Thanks in advance
