2

I have server with CentOS 6.2, apache 2.2, php 5.3, mysql 5.1. Server is installed in KVM container.

From time to time apache stalls. It just stops accepting connections. Looks like it reaches ServerLimit and keep accepted connections opened.

The strange thing in my opinion is that after service httpd restart server goes down again very fast. It accepts connections only for 1-2 seconds may be. And stalls again. But after reboot it can work several hours without problems. Or can stall again in several minutes. But it is not seconds like after service httpd restart.

Another noticeable thing is that after server stalled, I'm not able to connect to it via ssh. But if I already was connected, I can type commands.. but, if I type i.e. service httpd restart system performs my command very slow (but CPU is free and memory is rather free). So, it looks like server cant allocate some resource and waits for it. But I'm not sure what exactly it waits for..

there are variables from sysctl -a, which were changing during server working

servername/vars/

Server settings

ServerLimit 140
MaxClients 128
MinSpareServers 5
MaxSpareServer 10
MaxRequestsPerChild 4000
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 10

Setting keepAlive to off didnt help

update

I think i know what is wrong. Looks like host's SELinux by some reasons randomly blocks kvm-guest's writing to disk. May be it is because i didnt use the libvirt pool, but just lvm-volumes to add disks devices to guest. now I need only wait while it will happen again and check logs to be sure.

update 2

NO, it is not SELinux. Question is still opened. Any ideas?

Andrey
  • 121
  • 2
  • What is that `http://174.36.251.141/vars/` link? Also, I'd suggest looking into possible networking issues that may cause this. Perhaps capturing some packets just to make sure it isn't network related. – Belmin Fernandez Mar 10 '12 at 19:54
  • _What is that http://174.36.251.141/vars/ link?_ – Andrey Mar 10 '12 at 20:37
  • Hm.. yes that is what I asked :-) – Belmin Fernandez Mar 10 '12 at 20:41
  • :) _What is that http://174.36.251.141/vars/ link?_ it is the dynamics of system variables, if do `sysctl -a` + `netstat -n | grep -c WAIT` + `netstat -n | grep -c ESTAB`, but only for changing values (so, only those values which were changed during monitoring, will be shown). Every graph has the name. It is the name of sysctl variable (or TIME_WAIT and ESTABLISHED for netstat) _Perhaps capturing some packets just to make sure it isn't network related._ How to do it? I can say if i already connected via ssh, i can type after apache stalls. And httpd restart doesnt help, only reboot. – Andrey Mar 10 '12 at 20:45
  • also, i have access to host server and can check network from there. Network looks ok. I can connect to host at least. And some time ago it was another KVM container with the same problem but another ip address... and the same behaviour. So, i think it is not network. At least not external network. – Andrey Mar 10 '12 at 20:48
  • @BeamingMel-Bin just to be sure you know.. – Andrey Mar 10 '12 at 20:58

2 Answers2

0

Try lowering: MinSpareServers 5 and MaxSpareServer 10. Also tell us how much RAM do you have ?

Spacedust
  • 558
  • 5
  • 12
  • 28
0

This was a problem with iptables. I'm not sure what was it exactly and why... But as fast as I have moved rule -A INPUT -i lo -j ACCEPT to the top of all other rules, it started work correctly and no this problem again ever (before this the first rule was about eth0).

Andrey
  • 121
  • 2
  • If you suspect it is iptables' problem, could you post the iptables rules? – Raymond Tau Dec 25 '13 at 16:41
  • It was about 2 years ago from now. I dont remember the details. But I can say that the first rule was like this: `-A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT`, and now the first rule is `-A INPUT -i lo -j ACCEPT`. Also one more detail, apache was behind nginx. So, nginx has listened main ip address and apache listened only 127.0.0.1. And nginx was sending php requests to apache and served static files itself. – Andrey Dec 26 '13 at 06:19