We have a Windows 7 Pro machine running Apache/php/postgres processing Ajax requests under a constant load (several every second). Its also running various other applications, which perform lots of disk writes.
Usually Ajax responses are received in under a second, but occasionally (~once in a 24h period) for a period of upto 15 seconds no responses are sent, and then they are all sent at the end, i.e. it appears the server is blocked for upto 15 seconds. This causes Ajax to timeout on the client side.
Logs from Apache and other applications back this up. Perfmon shows various counters drop to zero/near-zero - HD activity, CPU activity, network activity, etc. httpd#1 seems to be the only process which still has some CPU activity, albeit reduced.
How can I determine the cause of the hang? Can perfmon or another tool tell me what resource is blocking? (Is the 'Windows Performance Toolkit' or 'Process Monitor' any good for this?)
NB Apache has ample threads, postgres ample connections, CPU and RAM are not maxed out, and we've tried power options, drivers, sfc /scannow, chkdsk /r, memtest, etc.
Update 22/03/2013 10:26:
Thanks for all your responses so far. More information:
Hardware:
- Chassis: Westek 2U Rack Mount Motherboard: Intel Q35 1333FSB (5xPCI, 2xPCI-E, SATA II I/F, VGA I/F, 2xRS232, etc)
- RAM: 2x 2GB DDR2 PC2-5300 non-ECC CL4 240 pin Memory Module (3GB usable as 32-bit OS)
- Processor: Intel Core2 Quad Q9550.2.83GHz 1066FSB 12MB Cache
- Storage: 2x Hitachi 320GB SATA 16MB Cache 7200 NCQ in SATA-II RAID Box - Intel Raid 1, NTFS
- Power: 2x 400W PSU - dual redundant
- Modem: StarTech external v.92 56k USB Fax Modem
- PCI card: Telephony card
OS:
- Windows 7 Pro SP1 32-bit
Advanced Performance Options:
(System Properties > Advanced > Performance > Settings > Advanced)
- Processor scheduling: best performance for programs
- Virtual memory: Automatically manage paging file size for all drives
- Total paging size for all drives:
- Minimum allowed: 16 MB
- Recommended: 4591 MB
- Currently allocated: 3061 MB
Update 22/03/2013 11:46:
Screenshot from perfmon:
http://i46.tinypic.com/fndyit.png (I don't have enough reputation to embed it in the post)
The period during which the server is unresponsive is 07:44:15 - 07:44:22 - whilst the CPU drops below 20%. (NB this is from another server with weaker CPU and older unoptimized software - usually CPU is not this high!)
Update 04/04/2013 16:53:
We found the culprit - the HDD. Only took a month!
How we got there:
Process Monitor confirmed that the disk was blocking on all writes during the incidents. We first tried updating the RAID drivers. This improved things - the CPU,etc wouldn't completely drop to zero, but the disk was still blocking. We then tried disabling RAID - this had no effect. We tried reducing the disk usage by disabling various logging and this helped. We then tried swapping the HDD for another (of lower spec), using the image from the first, and the problem completely disappeared.
So what was wrong with our HDD?
The disk we were using was a "Hitachi TravelStar 7k500 (Enhanced Availability variant)". It appears that the duty cycle has been limited to ensure ‘Enhanced Availability’ for this model, which may not suitable for particularly heavy disk usage. According to Resource Monitor, our disk usage is around 400KB/sec.