Linux/Apache performance very slow even on local network

Question

I have an Ubuntu server machine running Apache and MYSQL. System and version info is as follows: Linux kernel 3.0.0.-12 Apache/2.2.20 MySQL Ver 14.14.Distrib 5.1.58

I am running a few websites on this server, some HTML only, some PHP/MySQL. THe [problem is that response time is very slow, both on static as well as the dynamic sites. Sometimes it takes more than 10 seconds before a response is given, this makes the sites very slow and almost unusable.
The problem occurs even when requesting from the local network. I have added the involved subdomains to my /etc/hosts file, and abolve all the problem is not solved by using IP numbers instead of URL's. So there is no DNS lookup issue.
I have modified the log format by showing the response times and sometimes a files takes 12 seconds to be served, see the jquery~.js file in the example screenshot.
I have no explanation for this extremely long response time, but is is not even the only issue here, some other files takes a long time to be served too, but do not show a long response time in the log file.
So probably different tissues are involved here. I cannot find a solution until now, any suggestions??? THanx in advance, Klaas

link to screenshot picture from access logfile

Some extra configuration info:

apache2.conf (comment is removed)

LockFile ${APACHE_LOCK_DIR}/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_event_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

AccessFileName .htaccess

<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

DefaultType text/plain
HostnameLookups Off

ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn

Include mods-enabled/*.load
Include mods-enabled/*.conf
Include httpd.conf
Include ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Include conf.d/
Include sites-enabled/

And the virtual hostfile for one of the slow sites, in fact it is pretty straightforward...

<VirtualHost *:80>
    ServerAdmin blablabla@gmail.com
        ServerSignature EMail
        ServerName toenjoy.drsklaus.nl
    DocumentRoot /var/www/toenjoy.drsklaus.nl
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/toenjoy.drsklaus.nl/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride AuthConfig
                AuthType Basic
                AuthName "To Enjoy"
                AuthUserFile /etc/.htpasswd
                Require user petraaa
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn
    CustomLog /var/log/apache2/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>
</VirtualHost>

And the output of free -m:

klaas@ubuntu-server:/etc/apache2$ free -m
             total       used       free     shared    buffers     cached
Mem:          1997       1401        595          0        144       1017
-/+ buffers/cache:        238       1758
Swap:         2035          0       2035

and I have no indication that swapping occurs on the moments the site is slow. I have runned top and it does not appear to be a CPU issue. I have the impression that the spawning of a apache thread could maybe be the bottleneck but it is just a suggestion. Maybe this gives some extra information!

EDIT: The problem seemed to be gone for some time but occurs again! And not only with Apache, also connecting using SSH takes a tremendous time, sometimes it takes up to 15 seconds before the keyphrase is asked for. Also scp works very slowly. The behavious is really unpredoctable and makes the server very hard to use. Any ideas...?

Post the output of diagnostic tools such `top`, `free -m` during the slow performance ... maybe the system is swapping? running low on CPU? memory? Also, post relevant portions of your httpd.conf (particularly, `StartServers, MinSpareServers, MaxSpareServers, ServerLimit, MaxClients, MaxRequestsPerChild`) and vhost configs. This should provide a good starting point for someone to assist you further. — KM., Feb 13 '12 at 15:10
How many cores does the CPU have? What does the CPU usage look like? — David Schwartz, Nov 02 '12 at 07:15

score 1 · Answer 1 · edited Sep 14 '15 at 18:17

Edit: Try these one at a time!

A couple of things jump out:

MaxRequestsPerChild 0

If you have no reason to use 0 (process never dies) then increase it by a little (default is 1000) to say 100 and restart, and see if that gets you improvements. May prevent memory leaks.

MaxClients 150

Depending on how much memory your server has, and if all three modules are being called at at the same time, you may run out of memory. For e.g., if your apache process is 20MB in size, you will have 150 Clients x 20MB x 3 mods = 9000 MB ~ 8.8 GB. Even if only one of them is being called, then you are using up ~ 2.9 GB.

If you are not using all three mods, uninstall the unneeded ones from your apache instance.

~~Also, how about the output of top and free -m during this slow performance?~~

I will check it out, still not sure which of the 3 mods are actually used. Try to figure that out first... — klausch, Feb 13 '12 at 19:33

score 1 · Answer 2 · answered Mar 08 '13 at 21:06

What happens when you attempt to access via lynx or curl from the local host or from a machine on the network?

Also, look at this block from Apache:

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
    Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

I would recommend getting rid of that whole block, restarting Apache & trying again. I have had odd hangs as well when Apache does some deny actions against a localhost or a local IP address. Weird DNS lookup issues can hang Apache on a per request basis. And a delay of 12 seconds per page makes me think that is the cause. But it's not 100% clear from your config if the block I am pointing out is the culprit or not. But do some kind of search for DNS lookup related tasks connected to Apache.

Also, it's hard to say what settings should be on your server without observing your sites real traffic, but I suggest that your KeepAlive settings seem resource heavy. I would recommend changing as so:

KeepAlive On
MaxKeepAliveRequests 50
KeepAliveTimeout 2

I would also recommend lowering MaxClients and adding ServerLimit adjusting MaxRequestsPerChild as well in your main apache2.conf like so:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    ServerLimit          90
    MaxClients           90
    MaxRequestsPerChild 2000
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      64 
    ThreadLimit          64
    ThreadsPerChild      25
    ServerLimit          90
    MaxClients           90
    MaxRequestsPerChild 2000
</IfModule>

<IfModule mpm_event_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      64 
    ThreadLimit          64
    ThreadsPerChild      25
    ServerLimit          90
    MaxClients           90
    MaxRequestsPerChild 2000
</IfModule>

Also, I highly recommend installing Munin on your machine to get a grip on resource usage. Saves a lot of headaches & gives a good broad perspective on things.

score 1 · Answer 3 · answered Nov 04 '14 at 02:18

Apache is a complex beast and may be the problem here...or may not. I would start debugging the general network slowness before reaching into that dark hole. Specifically:

Try running netcat in server mode on a port, configured to hand out a local file's contents, and then connecting to it from the machine itself. For example:

while true ; do nc -l 80 < index.html ; done

That eliminates apache, the physical network, and DNS or IP link issues. Note that the file served is local - no NFS mounts for example.

Same again, but connect over the local network from the same subnet
Again, but from a different subnet

If you get this far, your server is good, and your network routing is OK too.

Now clean up DNS, and make quite sure that reverse lookups work. dig 4.3.2.1.in-addr.arpa for example to test the lookup of the 1.2.3.4 address. If it's slow, that's a problem and you'll need to noodle about with /etc/resolv.conf, dnsmasq or whatever you have. Don't proceed until this is rock solid. Worth trying SSH/SCP into the box at this point too - it should be fast.

Next, Apache. If you make it this far you know your network is good, DNS works and the box is OK. So that leaves your application (ie the www server). Try to strip down the config to the bare minimum and see how you get on. For example here is one.

Good luck!

Linux/Apache performance very slow even on local network

3 Answers3