many graceful restarts in httpd error log?

Question

Our server was down , and we restatred the services (nginx & httpd), and when i look at the logs, i've found these lines , there are so many Graceful restart requested, doing restart lines, whats wrong? i have 100's of lines like below

[Tue Nov 10 03:35:12.421652 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Tue Nov 10 03:35:12.421669 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
JPEG datastream contains no image
JPEG datastream contains no image
JPEG datastream contains no image
Empty input file
[Wed Nov 11 03:37:09.103375 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:37:09.168114 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:37:09.168701 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:37:09.294548 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:37:09.294563 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:37:26.407712 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:37:26.468105 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:37:26.468841 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:37:26.587135 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:37:26.587148 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:37:26.594527 2015] [core:error] [pid 60224] AH00546: no record of generation 48 of exiting child 100499
[Wed Nov 11 03:37:38.603784 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:37:38.660377 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:37:38.660946 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:37:38.786062 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:37:38.786076 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:37:48.571168 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:37:48.649259 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:37:48.650033 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:37:48.778361 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:37:48.778375 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:38:04.996390 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:38:05.064130 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:38:05.064883 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:38:05.192110 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:38:05.192122 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:38:20.318824 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:38:20.389300 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:38:20.390050 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:38:20.540335 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:38:20.540351 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 03:38:33.263314 2015] [mpm_prefork:notice] [pid 60224] AH00171: Graceful restart requested, doing restart
[Wed Nov 11 03:38:33.328249 2015] [auth_digest:notice] [pid 60224] AH01757: generating secret for digest authentication ...
[Wed Nov 11 03:38:33.328853 2015] [lbmethod_heartbeat:notice] [pid 60224] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 03:38:33.446190 2015] [mpm_prefork:notice] [pid 60224] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 03:38:33.446203 2015] [core:notice] [pid 60224] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Nov 11 19:01:38.520827 2015] [suexec:notice] [pid 3902] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Nov 11 19:01:38.561249 2015] [auth_digest:notice] [pid 3902] AH01757: generating secret for digest authentication ...
[Wed Nov 11 19:01:38.561959 2015] [lbmethod_heartbeat:notice] [pid 3902] AH02282: No slotmem from mod_heartmonitor
[Wed Nov 11 19:01:38.705993 2015] [mpm_prefork:notice] [pid 3902] AH00163: Apache/2.4.6 (CentOS) PHP/5.6.7 mod_apreq2-20090110/2.8.0 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Wed Nov 11 19:01:38.706025 2015] [core:notice] [pid 3902] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Nov 12 03:45:09.304011 2015] [mpm_prefork:notice] [pid 3902] AH00171: Graceful restart requested, doing restart
[Thu Nov 12 03:45:09.361104 2015] [auth_digest:notice] [pid 3902] AH01757: generating secret for digest authentication ...

Any cron that might trigger that ? Is it a regular CentOS that you're using ? What about different days ? Is it always on the same hours ? You can check the syslog at the same time to see if anything is triggered at the same time — Pierre-Alain TORET, Nov 12 '15 at 13:00
@pat thanks for the reply , no crons were triggered , im using centos7 , its not happening eveyday, syslogs are not enabled :( — ɹɐqʞɐ zoɹǝɟ, Nov 13 '15 at 16:35
any news ? did you check the system logs with `journalctl` ? — Pierre-Alain TORET, Nov 24 '15 at 13:54

score 10 · Accepted Answer · answered Feb 26 '16 at 10:10

10

okay, i've found the reason , these graceful restarts are happening because of log rotations (crond gracefully restart the http service after log rotation) some logs are getting rotated very frequently (because of some misconig). Thats all

answered Feb 26 '16 at 10:10

ɹɐqʞɐ zoɹǝɟ

277
3
9

Looks like the frequent logrotate restart calls are filling your logs and causing the logrotations. Same for mine, but I severely need my logs now, and errors.log.14.gz goes back a whopping 45 minutes. All the useful stuff has been long since destroyed. Did you ever resolve the issue on your system? Thanks. – Nate T Jan 24 '22 at 02:28

many graceful restarts in httpd error log?

1 Answers1