History:
We moved a Codeigniter 3 Installation from Bluehost to a T3.2xlarge. That single instance is hosting apache2 and a mysql server as a local database.
On Bluehost, the instance was running fine, migration was done since Bluehost itself had outages and we wanted more reliable hosting.
Error
Since the migration the Page is randomly going down completely. Trying to restart apache2 with:
sudo service apache2 restart
Does not work, it requires a full reboot of the EC2 instance to get the service running again. After rebooting EC2, apache2 and mysql is running and the page is up without starting the services after the reboot of the instance.
Debug attempt 1
Since the page went down when database intense crons were run, I assumed the mysql server was the bottleneck. Migrating the full database into a serverless RDS should eliminate all database related bottlenecks. The same database intense crons are finishing now. To further eliminate the cron being the reason for the system going down, I cloned the EC2 and used the clone to run the cron while the original hosts the webpage the domain is pointing to.
However, random outages still persist.
Debug attempt 2
Assuming it is a memory issue, after checking phpinfi.php I saw that PHP had 128Mb of RAM ( on a 32Gb machine ), so just to see if more RAM helps:
- memory_limit set to 8192Mb
- reboot the EC2
- restart php7.4-fpm
- restart apache2
phpinfo confirmed the memory_limit is set to 8192M.
Random outages still persist.
Debug attempt 3
Checking the command:
sudo apache2ctl -t
returns:
Syntax OK
Checking the command:
nano /var/log/apache2/error.log
contains:
[mpm_worker:notice] AH00295: caught SIGTERM, shutting down
So I assume that Apache somehow is shutting down for some reason but unable to restart.
Checking the command:
sudo service apache2 restart
does not throw errors
Checking the command:
sudo apache2ctl restart
does not throw errors
Checking the command:
/usr/sbin/apache2 -V
shows:
[core:warn] [pid 24560] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 81 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be
a valid directory, absolute or relative to ServerRoot
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2022-03-16T16:52:53
Server's Module Magic Number: 20120211:88
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM:
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/apache2"
-D SUEXEC_BIN="/usr/lib/apache2/suexec"
-D DEFAULT_PIDLOG="/var/run/apache2.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="mime.types"
-D SERVER_CONFIG_FILE="apache2.conf"
Where I can see 2 things:
- there is a issue with ${APACHE_RUN_DIR}
- Server MPM does not return a MPM
Checking the command:
apache2 -l
returns:
Compiled in modules:
core.c
mod_so.c
mod_watchdog.c
http_core.c
mod_log_config.c
mod_logio.c
mod_version.c
mod_unixd.c
What does not show a MPM module.
Checking the command:
apache2 -l
apache2ctl -l
returns:
Compiled in modules:
core.c
mod_so.c
mod_watchdog.c
http_core.c
mod_log_config.c
mod_logio.c
mod_version.c
mod_unixd.c
Checking the command:
a2query -M
returns:
worker
Question:
And this is where I have been stuck now. Is there anything else I can check or read from the debug attempt 3 to see why apache stops/does not restart and requires a full server reboot?
