I have already worked through AWS Elastic Beanstalk - Apache is restarting constantly
Our elastic beanstalk instances are reporting the following message in the error_log
[Mon Jun 26 22:01:01.878892 2017] [mpm_prefork:notice] [pid 8595] AH00173: SIGHUP received. Attempting to restart
*** Error in (wsgi:wsgi) ': double free or corruption (out): 0x00007f564cced560 ***
sometimes the error sequence will look more like this:
[Tue Jun 27 00:01:01.215260 2017] [:error] [pid 6429] [remote XX.XXX.XX.195:29773] mod_wsgi (pid=6429): Exception occurred processing WSGI script '/opt/python/current/app/site/settings/wsgi/__init__.py'.
[Tue Jun 27 00:01:01.215320 2017] [:error] [pid 6429] [remote XX.XXX.XX.195:29773] OSError: failed to write data
[Tue Jun 27 00:01:01.222407 2017] [:error] [pid 6430] [remote XX.XXX.XX.60:53313] mod_wsgi (pid=6430): Exception occurred processing WSGI script '/opt/python/current/app/site/settings/wsgi/__init__.py'.
[Tue Jun 27 00:01:01.222460 2017] [:error] [pid 6430] [remote XX.XXX.XX.60:53313] OSError: failed to write data
[Tue Jun 27 00:01:04.554810 2017] [core:warn] [pid 8595] AH00045: child process 7614 still did not exit, sending a SIGTERM
[Tue Jun 27 00:01:04.554850 2017] [core:warn] [pid 8595] AH00045: child process 7615 still did not exit, sending a SIGTERM
[Tue Jun 27 00:01:05.555958 2017] [mpm_prefork:notice] [pid 8595] AH00173: SIGHUP received. Attempting to restart
*** Error in (wsgi:wsgi) ': double free or corruption (out): 0x00007f5640cae900 ***
*** Error in (wsgi:wsgi) ': double free or corruption (out): 0x00007f78649b7970 ***
This will go on, almost every hour. The common message being:
[Mon Jun 26 22:01:01.878892 2017] [mpm_prefork:notice] [pid 8595] AH00173: SIGHUP received. Attempting to restart
I looked for the mpm_prefork
module conf block... and there is not one, so all the defaults are being used.
I looked for the logrotation
command being executed by elastic beanstalk
/var/log/httpd/* {
size 10M
missingok
notifempty
rotate 5
sharedscripts
compress
dateext
dateformat -%s
create
postrotate
/sbin/service httpd reload > /dev/null 2>/dev/null || true
endscript
olddir /var/log/httpd/rotated
}
Pretty standard stuff. My understanding of reload
is it attempts a graceful restart...
I am able to manually trigger the error message by executing sudo apachectl -k restart
although I can not find where this would be run during the log rotation.
We have downstream services which appear to be throwing exceptions at the point this server hangs up all it's connections.
So my question is, what else could be causing the SIGHUP
within the mpm_prefork
during logrotate
? As far as I can tell, this should not be happening outside of an error condition.
Apache/2.4.18 (Amazon) mod_wsgi/3.5 Python/3.4.3