8

My server sometimes gets filled up with connections in apache stuck in a "Sending Reply" state, requiring me to restart apache. Most of the time this works, but sometimes I will get this error when trying to restart apache instead:

Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe"
for details.

Running systemctl status httpd.service or journalctl -xe as suggested then brings back the following relevant information:

Nov 15 06:24:06 hostname.biologyreporter.com systemd-logind[874]:
 Failed to remove runtime directory /run/user/1067: Device or resource busy

Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 [Fri Nov 15 06:24:27.255594 2019] [core:emerg] [pid 29509:tid 47498001208384]
 (28) No space left on device: AH00023: Couldn't create the mpm-accept mutex  
 
Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 (28) No space left on device: could not create accept mutex  
 
Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 AH00015: Unable to open logs

Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]:
 httpd.service: control process exited, code=exited status=1  
 
Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]:
 Failed to start Apache web server managed by cPanel EasyApache.
 -- Subject: Unit httpd.service has failed

It may then take ~10 minutes or so for apache to actually restart and the website to be functional again. I don't know why it is saying "No space left on device" as running df, for example, returns the following (nowhere is 100% of space taken):

Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs         3973024         0   3973024   0% /dev
tmpfs            3983400         0   3983400   0% /dev/shm
tmpfs            3983400    255296   3728104   7% /run
tmpfs            3983400         0   3983400   0% /sys/fs/cgroup
/dev/sda2      952008348 137586024 766039760  16% /
/dev/sda1         999320    134892    795616  15% /boot
/dev/loop0       3997376      8856   3778808   1% /tmp
tmpfs             796684         0    796684   0% /run/user/0
tmpfs             796684         0    796684   0% /run/user/1022

I've been unable to find any reliable information on the cause and solution of the specific error(s) as listed above

  • No space left on device: AH00023: Couldn't create the mpm-accept mutex and
  • Failed to remove runtime directory /run/user/1067: Device or resource busy

What should I do to solve this so that these errors don't show up and apache always restarts without issues?

PauloBoaventura
  • 133
  • 7
user548654
  • 81
  • 1
  • 1
  • 3

3 Answers3

15

This is most likely caused by the semaphore limit of the OS, and Apache not properly cleaning up after itself.

See more details about semaphore limits https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-setting_semaphores-setting_semaphore_parameters

Those errors mean that there is a lack of inner-process communication resources in the system, such as semaphores or shared memory segments. Try running next to see if the semaphore limit is reached and see results:

ipcs -s | wc -l

cat /proc/sys/kernel/msgmni

cat /proc/sys/kernel/sem

Execute the following command via SSH or console and see if it helps :

ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}'

ipcs -s lists currently used semaphores.

awk -v user=apache filters out the ones, owned by apache user and next part executes icprm -s "id" for each of it which removes that semaphore.

I.e. whole command removes all semaphores owned by apache user

dzup4uk
  • 191
  • 9
3

While the answer by @dzup4uk seems to be able to point out what the issue is, in order to actually solve the issue you have to ensure that the semaphores get cleaned up. ipcs command will show what the issue is but in order to actually solve the issue you have to do some cleaning which can be done with

ipcrm -a
Barnabas Busa
  • 712
  • 5
  • 10
  • Good tip here @Barnabas Busa. One small observation. On my system, I found semaphores being used by root associated with critical system processes. Perhaps it would be best to instead use `ipcrm -s ` so that a semaphore associated with a critical system process is not removed? – JeremyCanfield Aug 03 '21 at 05:56
  • Please, do not execute this command on production, as i just figured it out the command above stops apache - So obviously it will affect your clients, do at a time where no one is using it. – Raul Chiarella Jul 22 '22 at 17:34
0

If your Apache is running under the nobody user then you need to run it with nobody like so:

ipcs -s | awk -v user=nobody '$3==user {system("ipcrm -s "$2)}'
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
Louisvdw
  • 1