0

I'm having difficult times trying out to figure which are the problems Apache2 is having for it to die "randomly". I get noticed with some insults lol and have to restart it manually

Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-141-generic x86_64)
Server version: Apache/2.4.29 (Ubuntu)
Server built: 2020-08-12T21:33:25

Memory total used free shared buff/cache available
Mem: 2041 132 540 2 1369 1731
Swap: 1048 1 1047

Default Apache config:

   <VirtualHost *:80>
    ServerName SERVER_IP:80
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    <Directory /var/www/html>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            allow from all
            Require all granted
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

mpm_prefork.conf

<IfModule mpm_prefork_module>
StartServers        5
MinSpareServers     5
MaxSpareServers     10
MaxRequestWorkers   900
MaxConnectionsPerChild  0

The apache error.log file (last 4 days)

[Mon May 24 04:14:44.411885 2021] [:error] [pid 7932] [client PROXYPASS_IP:52274] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 04:34:28.497231 2021] [:error] [pid 7932] [client PROXYPASS_IP:57038] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 05:40:15.417989 2021] [:error] [pid 7931] [client PROXYPASS_IP:47890] script '/var/www/html/compliancepolicies.inc.php' not found or unable to stat
[Mon May 24 05:59:27.146680 2021] [:error] [pid 7932] [client PROXYPASS_IP:51980] script '/var/www/html/compliancepolicies.inc.php' not found or unable to stat
[Mon May 24 07:23:38.786537 2021] [:error] [pid 7933] [client PROXYPASS_IP:42934] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 07:23:40.291081 2021] [:error] [pid 7929] [client PROXYPASS_IP:42936] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 08:02:24.889646 2021] [:error] [pid 7930] [client PROXYPASS_IP:54024] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 08:02:26.436263 2021] [:error] [pid 8486] [client PROXYPASS_IP:54028] script '/var/www/html/index.php' not found or unable to stat
[Mon May 24 12:48:00.390667 2021] [:error] [pid 8486] [client PROXYPASS_IP:41238] script '/var/www/html/zabbix.php' not found or unable to stat
[Mon May 24 13:04:15.559852 2021] [:error] [pid 7933] [client PROXYPASS_IP:53794] script '/var/www/html/zabbix.php' not found or unable to stat
[Tue May 25 02:41:24.347577 2021] [:error] [pid 7929] [client PROXYPASS_IP:52622] script '/var/www/html/wp-config.php' not found or unable to stat
[Tue May 25 02:41:27.066550 2021] [:error] [pid 8486] [client PROXYPASS_IP:52632] script '/var/www/html/wp-config-sample.php' not found or unable to stat
[Tue May 25 04:26:30.536685 2021] [:error] [pid 7930] [client PROXYPASS_IP:42512] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 04:26:32.413864 2021] [:error] [pid 8486] [client PROXYPASS_IP:42522] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 05:47:46.793367 2021] [:error] [pid 7929] [client PROXYPASS_IP:54138] script '/var/www/html/index.php' not found or unable to stat
[Tue May 25 06:04:29.557875 2021] [:error] [pid 7932] [client PROXYPASS_IP:56360] script '/var/www/html/index.php' not found or unable to stat
[Tue May 25 07:50:38.303574 2021] [:error] [pid 7931] [client PROXYPASS_IP:48612] script '/var/www/html/wp-config.php' not found or unable to stat
[Tue May 25 07:50:41.045859 2021] [:error] [pid 7933] [client PROXYPASS_IP:48620] script '/var/www/html/wp-config-sample.php' not found or unable to stat
[Tue May 25 09:09:20.992386 2021] [:error] [pid 7929] [client PROXYPASS_IP:35758] script '/var/www/html/search.php' not found or unable to stat
[Tue May 25 09:17:50.675332 2021] [:error] [pid 8486] [client PROXYPASS_IP:39500] script '/var/www/html/lfm.php' not found or unable to stat
[Tue May 25 09:25:46.620318 2021] [:error] [pid 7931] [client PROXYPASS_IP:42236] script '/var/www/html/search.php' not found or unable to stat
[Tue May 25 09:33:42.456898 2021] [:error] [pid 7932] [client PROXYPASS_IP:44744] script '/var/www/html/lfm.php' not found or unable to stat
[Tue May 25 09:42:43.906866 2021] [:error] [pid 7929] [client PROXYPASS_IP:47104] script '/var/www/html/admin.php' not found or unable to stat
[Tue May 25 09:42:46.670982 2021] [:error] [pid 8486] [client PROXYPASS_IP:47114] script '/var/www/html/user.php' not found or unable to stat
[Tue May 25 10:37:47.776532 2021] [:error] [pid 7930] [client PROXYPASS_IP:43198] script '/var/www/html/admin.php' not found or unable to stat
[Tue May 25 10:37:50.676811 2021] [:error] [pid 7931] [client PROXYPASS_IP:43216] script '/var/www/html/user.php' not found or unable to stat
[Tue May 25 11:33:54.751108 2021] [:error] [pid 7931] [client PROXYPASS_IP:48124] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 11:47:39.089285 2021] [:error] [pid 7929] [client PROXYPASS_IP:55144] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 11:52:15.245326 2021] [:error] [pid 7929] [client PROXYPASS_IP:57810] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 12:05:51.302652 2021] [:error] [pid 8486] [client PROXYPASS_IP:51014] script '/var/www/html/wp-login.php' not found or unable to stat
[Tue May 25 13:27:59.278228 2021] [:error] [pid 8486] [client PROXYPASS_IP:44234] script '/var/www/html/email_passthrough.php' not found or unable to stat
[Tue May 25 13:41:58.115424 2021] [:error] [pid 7932] [client PROXYPASS_IP:50026] script '/var/www/html/email_passthrough.php' not found or unable to stat
[Tue May 25 15:41:46.967467 2021] [:error] [pid 7932] [client PROXYPASS_IP:48632] script '/var/www/html/index.php' not found or unable to stat
[Tue May 25 15:56:29.117296 2021] [:error] [pid 8486] [client PROXYPASS_IP:55530] script '/var/www/html/index.php' not found or unable to stat
[Tue May 25 21:24:04.682693 2021] [:error] [pid 7930] [client PROXYPASS_IP:48692] script '/var/www/html/login_up.php' not found or unable to stat
[Tue May 25 21:38:45.465703 2021] [:error] [pid 7931] [client PROXYPASS_IP:60814] script '/var/www/html/login_up.php' not found or unable to stat
[Wed May 26 01:38:24.515955 2021] [:error] [pid 7933] [client PROXYPASS_IP:39402] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 01:57:16.951116 2021] [:error] [pid 7930] [client PROXYPASS_IP:45162] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 04:38:19.462977 2021] [:error] [pid 7931] [client PROXYPASS_IP:46570] script '/var/www/html/snippets.inc.php' not found or unable to stat
[Wed May 26 04:57:06.747394 2021] [:error] [pid 7932] [client PROXYPASS_IP:48582] script '/var/www/html/snippets.inc.php' not found or unable to stat
[Wed May 26 06:42:23.204774 2021] [:error] [pid 7931] [client PROXYPASS_IP:38488] script '/var/www/html/php.php' not found or unable to stat
[Wed May 26 06:42:24.631918 2021] [:error] [pid 7932] [client PROXYPASS_IP:38492] script '/var/www/html/phpinfo.php' not found or unable to stat
[Wed May 26 06:42:25.995827 2021] [:error] [pid 7933] [client PROXYPASS_IP:38496] script '/var/www/html/info.php' not found or unable to stat
[Wed May 26 06:42:27.355651 2021] [:error] [pid 7929] [client PROXYPASS_IP:38498] script '/var/www/html/infophp.php' not found or unable to stat
[Wed May 26 06:42:28.749061 2021] [:error] [pid 7930] [client PROXYPASS_IP:38502] script '/var/www/html/php_info.php' not found or unable to stat
[Wed May 26 06:42:30.105537 2021] [:error] [pid 8486] [client PROXYPASS_IP:38504] script '/var/www/html/test.php' not found or unable to stat
[Wed May 26 06:42:31.466479 2021] [:error] [pid 7931] [client PROXYPASS_IP:38512] script '/var/www/html/i.php' not found or unable to stat
[Wed May 26 06:42:32.843191 2021] [:error] [pid 7932] [client PROXYPASS_IP:38518] script '/var/www/html/asdf.php' not found or unable to stat
[Wed May 26 06:42:34.203376 2021] [:error] [pid 7933] [client PROXYPASS_IP:38566] script '/var/www/html/pinfo.php' not found or unable to stat
[Wed May 26 06:42:35.582707 2021] [:error] [pid 7929] [client PROXYPASS_IP:38574] script '/var/www/html/phpversion.php' not found or unable to stat
[Wed May 26 06:42:36.931655 2021] [:error] [pid 7930] [client PROXYPASS_IP:38578] script '/var/www/html/time.php' not found or unable to stat
[Wed May 26 06:42:38.387809 2021] [:error] [pid 8486] [client PROXYPASS_IP:38584] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 06:42:39.743931 2021] [:error] [pid 7931] [client PROXYPASS_IP:38586] script '/var/www/html/temp.php' not found or unable to stat
[Wed May 26 06:42:41.143446 2021] [:error] [pid 7932] [client PROXYPASS_IP:38594] script '/var/www/html/old_phpinfo.php' not found or unable to stat
[Wed May 26 06:42:42.485594 2021] [:error] [pid 7933] [client PROXYPASS_IP:38598] script '/var/www/html/infos.php' not found or unable to stat
[Wed May 26 06:42:43.832260 2021] [:error] [pid 7929] [client PROXYPASS_IP:38600] script '/var/www/html/linusadmin-phpinfo.php' not found or unable to stat
[Wed May 26 06:42:45.203621 2021] [:error] [pid 7930] [client PROXYPASS_IP:38606] script '/var/www/html/php-info.php' not found or unable to stat
[Wed May 26 07:33:26.421135 2021] [:error] [pid 8486] [client PROXYPASS_IP:55784] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 07:52:46.969891 2021] [:error] [pid 7932] [client PROXYPASS_IP:40388] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 12:06:27.502320 2021] [:error] [pid 7929] [client PROXYPASS_IP:37278] script '/var/www/html/php.php' not found or unable to stat
[Wed May 26 12:06:28.873814 2021] [:error] [pid 7930] [client PROXYPASS_IP:37334] script '/var/www/html/phpinfo.php' not found or unable to stat
[Wed May 26 12:06:30.206625 2021] [:error] [pid 9596] [client PROXYPASS_IP:37408] script '/var/www/html/info.php' not found or unable to stat
[Wed May 26 12:06:31.590073 2021] [:error] [pid 8486] [client PROXYPASS_IP:37448] script '/var/www/html/infophp.php' not found or unable to stat
[Wed May 26 12:06:32.932107 2021] [:error] [pid 7931] [client PROXYPASS_IP:37490] script '/var/www/html/php_info.php' not found or unable to stat
[Wed May 26 12:06:34.306775 2021] [:error] [pid 7932] [client PROXYPASS_IP:37516] script '/var/www/html/test.php' not found or unable to stat
[Wed May 26 12:06:35.701149 2021] [:error] [pid 9594] [client PROXYPASS_IP:37632] script '/var/www/html/i.php' not found or unable to stat
[Wed May 26 12:06:37.068345 2021] [:error] [pid 7933] [client PROXYPASS_IP:37710] script '/var/www/html/asdf.php' not found or unable to stat
[Wed May 26 12:06:38.426905 2021] [:error] [pid 7929] [client PROXYPASS_IP:37810] script '/var/www/html/pinfo.php' not found or unable to stat
[Wed May 26 12:06:39.855108 2021] [:error] [pid 7930] [client PROXYPASS_IP:37902] script '/var/www/html/phpversion.php' not found or unable to stat
[Wed May 26 12:06:41.295390 2021] [:error] [pid 9596] [client PROXYPASS_IP:38004] script '/var/www/html/time.php' not found or unable to stat
[Wed May 26 12:06:42.634024 2021] [:error] [pid 8486] [client PROXYPASS_IP:38108] script '/var/www/html/index.php' not found or unable to stat
[Wed May 26 12:06:44.767269 2021] [:error] [pid 7931] [client PROXYPASS_IP:38222] script '/var/www/html/temp.php' not found or unable to stat
[Wed May 26 12:06:46.129798 2021] [:error] [pid 7932] [client PROXYPASS_IP:38346] script '/var/www/html/old_phpinfo.php' not found or unable to stat
[Wed May 26 12:06:47.486713 2021] [:error] [pid 9594] [client PROXYPASS_IP:38442] script '/var/www/html/infos.php' not found or unable to stat
[Wed May 26 12:06:48.917990 2021] [:error] [pid 7933] [client PROXYPASS_IP:38534] script '/var/www/html/linusadmin-phpinfo.php' not found or unable to stat
[Wed May 26 12:06:50.382427 2021] [:error] [pid 7929] [client PROXYPASS_IP:38598] script '/var/www/html/php-info.php' not found or unable to stat
[Wed May 26 20:32:46.990524 2021] [:error] [pid 9596] [client PROXYPASS_IP:33452] script '/var/www/html/getcfg.php' not found or unable to stat
[Wed May 26 20:52:09.915084 2021] [:error] [pid 7932] [client PROXYPASS_IP:50880] script '/var/www/html/getcfg.php' not found or unable to stat
[Fri May 28 17:11:34.033543 2021] [core:warn] [pid 7924] AH00045: child process 7929 still did not exit, sending a SIGTERM
[Fri May 28 17:11:34.033689 2021] [core:warn] [pid 7924] AH00045: child process 7930 still did not exit, sending a SIGTERM
[Fri May 28 17:11:34.033703 2021] [core:warn] [pid 7924] AH00045: child process 7931 still did not exit, sending a SIGTERM
[Fri May 28 17:11:34.033716 2021] [core:warn] [pid 7924] AH00045: child process 7932 still did not exit, sending a SIGTERM
[Fri May 28 17:11:34.033728 2021] [core:warn] [pid 7924] AH00045: child process 7933 still did not exit, sending a SIGTERM
[Fri May 28 17:11:34.033740 2021] [core:warn] [pid 7924] AH00045: child process 8486 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.035954 2021] [core:warn] [pid 7924] AH00045: child process 7929 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.035995 2021] [core:warn] [pid 7924] AH00045: child process 7930 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.036001 2021] [core:warn] [pid 7924] AH00045: child process 7931 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.036008 2021] [core:warn] [pid 7924] AH00045: child process 7932 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.036014 2021] [core:warn] [pid 7924] AH00045: child process 7933 still did not exit, sending a SIGTERM
[Fri May 28 17:11:36.036021 2021] [core:warn] [pid 7924] AH00045: child process 8486 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038163 2021] [core:warn] [pid 7924] AH00045: child process 7929 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038217 2021] [core:warn] [pid 7924] AH00045: child process 7930 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038229 2021] [core:warn] [pid 7924] AH00045: child process 7931 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038241 2021] [core:warn] [pid 7924] AH00045: child process 7932 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038252 2021] [core:warn] [pid 7924] AH00045: child process 7933 still did not exit, sending a SIGTERM
[Fri May 28 17:11:38.038264 2021] [core:warn] [pid 7924] AH00045: child process 8486 still did not exit, sending a SIGTERM
[Fri May 28 17:11:40.040476 2021] [core:error] [pid 7924] AH00046: child process 7929 still did not exit, sending a SIGKILL
[Fri May 28 17:11:40.044186 2021] [core:error] [pid 7924] AH00046: child process 7930 still did not exit, sending a SIGKILL
[Fri May 28 17:11:40.046559 2021] [core:error] [pid 7924] AH00046: child process 7931 still did not exit, sending a SIGKILL
[Fri May 28 17:11:40.048150 2021] [core:error] [pid 7924] AH00046: child process 7932 still did not exit, sending a SIGKILL
[Fri May 28 17:11:40.050355 2021] [core:error] [pid 7924] AH00046: child process 7933 still did not exit, sending a SIGKILL
[Fri May 28 17:11:40.052198 2021] [core:error] [pid 7924] AH00046: child process 8486 still did not exit, sending a SIGKILL
[Fri May 28 17:11:41.055832 2021] [mpm_prefork:notice] [pid 7924] AH00169: caught SIGTERM, shutting down
[Fri May 28 17:11:41.230953 2021] [mpm_prefork:notice] [pid 819] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Fri May 28 17:11:41.230989 2021] [core:notice] [pid 819] AH00094: Command line: '/usr/sbin/apache2'
[Fri May 28 17:18:43.024640 2021] [mpm_prefork:notice] [pid 819] AH00169: caught SIGTERM, shutting down

There's a nginx server which acts as a proxypass (I guess) that throws 502 bad gateway when Internet client tries to use the webservice while Apache is dead

Any clues? Thanks a lot

Enrique Becerra
  • 145
  • 2
  • 8
  • Where did you get the mem stats? Very low figures, if you ask me. – Gerard H. Pille May 29 '21 at 03:50
  • What is the output of `dmesg` after Apache has crashed? – Tero Kilkanen May 29 '21 at 08:02
  • @GerardH.Pille with the command sudo free --mega . How much ram would you recommend me to ask for ? – Enrique Becerra May 29 '21 at 23:37
  • @TeroKilkanen server hasn't crashed yet, I guess due we're on weekend, but executing dmesg gave me the following https://i.imgur.com/1Eg2JX1.jpg – Enrique Becerra May 29 '21 at 23:41
  • 2Gb isn't a lot, probably not enough to handle 900 requestworkers. How much memory does one of the Apache servers take? – Gerard H. Pille May 30 '21 at 02:42
  • Why do you log the fact that you block multicast? To fill your disks? That jpeg was very informative. – Gerard H. Pille May 30 '21 at 02:51
  • @GerardH.Pille the 900 requestworkers number got from another question suggesting that value, but have no idea what it means sorry. I also don't know anything about that UFW log that dmesg output gave. As you can see I'm not very experienced in server mgmt – Enrique Becerra May 31 '21 at 12:13
  • 1
    Reduce the 900 to something your system can handle, take 5Mb per worker and make sure enough is left for the rest of your OS + apps + ... Then activate Apache's mod_status so that you can check if something isn't blocking the workers preventing them from answering requests. If you are only serving static pages, that probably won't be a problem, but an error in the nginx config might keep connections busy - again unlikely. Then check how you can disable useless logging. – Gerard H. Pille May 31 '21 at 13:02
  • Ok, I reduced 900 to 300, and enabled mod_status. I serve a codeigniter application (rest api) which connects to an Oracle Server for getting data. – Enrique Becerra May 31 '21 at 18:11
  • The Oracle server is on a separate system? Using persistent DB connections can boost performance - but not enough to allow for poor SQL or DB design. – Gerard H. Pille May 31 '21 at 20:06
  • @GerardH.Pille yes, Oracle is separate, and using persistent connections already – Enrique Becerra Jun 01 '21 at 22:55
  • Everything OK on the status page? A bit late I'm afraid: you may need "extendedstatus", if I remember correctly. – Gerard H. Pille Jun 01 '21 at 23:19
  • @GerardH.Pille yes, I guess it looks good and server still up. I've already enabled mod_status with ExtendedStatus On... https://i.imgur.com/nF7YqZt.jpg – Enrique Becerra Jun 03 '21 at 12:05
  • The image doesn't show an extended status, does it? I'm used to see each server. When I was webmaster, I used to save the status each 5 minutes, so that in case of problems, I could look back and find what started it. Eg. an URL that should be cached, an application that doesn't answer fast enough (bots love those). – Gerard H. Pille Jun 03 '21 at 12:14
  • @GerardH.Pille Yes, because I only took some part of the page, or I'd have to blur tons of ipaddresses. I'll try to cron -> create log files with status page and review that when it crashes again, nice tip – Enrique Becerra Jun 03 '21 at 18:01
  • "Tons of ipaddresses"?? "1 requests currently being processed, 9 idle workers" Your ipadresses weigh 100kg apiece? – Gerard H. Pille Jun 03 '21 at 19:53
  • @GerardH.Pille sorry, not "tons" but most of the time shows the reqs. of the other wan IPs which I should blur. But nothing strange in that list – Enrique Becerra Jun 04 '21 at 14:50

0 Answers0