PHP + Apache long wait time

Question

I've run into a bit of a brick wall with troubleshooting my dedicated websever. Recently, my website spiked with the amount of request/per second and it crashed.

The box original had 8GB of RAM, 8 Core Xeon E3-1230, 1TB 7,200 RPM Disk (No Raid), 100Mbit dedicated networking.

After the spike, I increased the RAM to 24GB in order to support more concurrent users.

Apache seems to handle well, even with 3000 concurrent users, it will return HTML and static content very quickly (un-cached).

To further test the differences between Apache/HTML and Apache/PHP, I ran ab.

Both test.html and test.php have exactly the same static content, the PHP does not call any includes and does not connect with MySQL.

HTML Test

ab -n 500 -c 50 http://www.~~.com/test.html

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      252  375 190.3    276    1399
Processing:   254  354 121.5    282     657
Waiting:      253  353 121.4    280     653
Total:        510  730 231.7    573    1675

PHP Test

ab -n 500 -c 50 http://www.~~~.com/test.php

Connect:      248  275  51.1    267    1316
Processing:   256 4167 6210.2   2262   41489
Waiting:      253 4166 6210.2   2262   41489
Total:        509 4442 6212.4   2523   41754

Pingdom also reports a long wait time when accessing a PHP script.

I'm getting a similar result on WebPageTest.org, although better, the first time to byte is F:

           Load Time   **First Byte**   Start Render    DOM Elements    Time    Requests    Bytes In    Time    Requests    Bytes In
First View  2.061s  **0.839s**  0.000s  55  2.061s  20  428 KB  2.061s  20  430 KB

Here is my top results:

I/O Test

Under heavy load, the wa% can increase to 95% for a few milliseconds.

I ran iostat during load:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.37    0.00    5.18    0.56    0.00   85.88

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00    45.50  3.00 48.00   136.00   748.00    17.33     3.05   59.76   2.53  12.90
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb3              0.00    45.50  3.00 48.00   136.00   748.00    17.33     3.05   59.76   2.53  12.90

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.00    0.00    3.56    0.69    0.00   91.75

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               6.00   118.50  9.50 21.00   996.00  1116.00    69.25     0.29    9.44   1.66   5.05
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb3              6.00   118.50  9.50 21.00   996.00  1116.00    69.25     0.29    9.44   1.66   5.05

To me, that doesn't look bad, but I may be missing something.

I'm using FCGI

<IfModule mod_fcgid.c>
FcgidMaxRequestLen 1547483648
FcgidMaxRequestInMem 52485760
FcgidIdleScanInterval 15000
FcgidBusyTimeout 15000
FcgidProcessLifeTime 7200
FcgidConnectTimeout 1800
FcgidIOTimeout 1800
PHP_Fix_Pathinfo_Enable 1
FcgidMaxRequestsPerProcess 1000
</IfModule>

And here is my Apache Conf (I'm using 2.4.x)

Timeout 60
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 10
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 15
</IfModule>
<IfModule itk.c>
MinSpareServers 5
MaxSpareServers 15
</IfModule>
ServerLimit 2200
MaxRequestWorkers 2000
MaxConnectionsPerChild 15000
KeepAlive On
KeepAliveTimeout 1
MaxKeepAliveRequests 2000

I've looked into my Apache Error Logs and Access Logs. Nothing strange to report.

I'm really scratching my head here.

I've tried turning off the firewall.

I've tried increasing Max Connections.

I've optimized mySQL and removing many slow queries (that were >0.5s).

What else can I do, is there anything I can use to help identify problems? Any help would be greatly appreciated.

P.S:

It's worth noting, that even when the server is being heavily accessed, PHPMyAdmin and cPanel are still very responsive. Nothing else seems to lag except the PHP on the website.

Answer 1

HTML file you're testing is just a plain file, all that Apache has to do is do a few system calls (open, read) and then serve its content.

PHP OTOH is actually quite a "heavy" option: it's an entire interpreter (bytecode compiler?). And since you're using concurrent testing (-c) who knows how well the requests are multiplexed to it? This does not have to be apache problem at all, but rather PHP's problem.

What I'd do:

1. Switch to Apache MPM (multi-threaded in multiple processes).

2. Do sequential test (no multiple concurrent requests), compare.

3. Run valgrind or some such on apache process and see where most CPU time is spent (apache or PHP).

4. run the same test but serving this page via nginx. Since nginx is based on async model and very fast, then if you get similar results, PHP is the culprit.

5. Finally, you could install Zend Optimizer (30-day trial is free) or smth like this: https://github.com/zendtech/ZendOptimizerPlus.

Really, comparing serving a static file to dynamically generated webpage is sort of apples to oranges comparison. None of the typical solutions (Python mod-apache, Django, PHP, etc) are going to be very fast in this regard, at least when compared to serving a static file. Node.js is an exception perhaps, due to sort of "low level" programming webpage directly in async model.

P.S. you have not quoted php.ini content. Post it and/or tweak it.

Answer 2

I think your problem is simply your hard disk with it's access times. Apache can cache html-pages in memory, but php-scripts are not cached; the need to be executed every time again. Therefore the php interpreter is called and it reads the script from the hdd. This takes a lot of time. The most slowest thing on your server is your HDD. On my pc, there is an extreme difference between applications starting from my SSD and my HDD (the SSD is up to 10 times faster!). If your HDD works, then the latency for executing an php script may increase drastically.

Possible solutions: get an SSD (maybe a small one just for often accessed data such as scripts) and minimize script calls (and HDD accesses) on your server. Make sure, the filesystem is defragmented (usually done automatically). If your php-script creates often the same contents, try caching them into an html file.

This answer could also help you: https://stackoverflow.com/questions/4181865/apache-php-caching