9

I have a website that uses https to transmit a javascript file to the client. The website is getsimpleapps.com.

It turns out that this file is loading 52 times slower with https (20.08s - 29.08s) that with http (380ms).

The homepage of the site shares the same slowness as the javacript file.

I've recently switched over from dreamhost to linode, and hacked at getting SSL to work on the new server until it did. I didn't do any crazy configuring.

The linode is running Ubuntu 12.04 and the site is on top of a (LAMP) stack.

My question to the stack overflow community is: How do I go about fixing SSL & HTTPS on my server? I know that stack overflow is littered with questions regarding the slowness of HTTPS but no real solutions are given. A ubuntu tutorial or configuration guide would be ideal.

file : /etc/apache2/sites-enabled/getsimpleapps.com

<VirtualHost *:80>
     ServerAdmin admin@getsimpleapps.com
     ServerName getsimpleapps.com
     ServerAlias www.getsimpleapps.com
     DocumentRoot /srv/sites/getsimpleapps.com/public/
     ErrorLog /srv/sites/getsimpleapps.com/logs/error.log
     CustomLog /srv/sites/getsimpleapps.com/logs/access.log combined
</VirtualHost>

<VirtualHost 50.116.58.18:443>
     SSLEngine On
     #SSLCertificateFile /etc/apache2/ssl/www.getsimpleapps.com.crt
     #SSLCertificateKeyFile /etc/apache2/ssl/www.getsimpleapps.com.key
     #SSLCACertificateFile /etc/apache2/ssl/comodo.crt
     SSLCertificateFile /etc/apache2/ssl/dreamhost/dh.crt
     SSLCertificateKeyFile /etc/apache2/ssl/dreamhost/dh.key
     SSLCACertificateFile /etc/apache2/ssl/dreamhost/dh.cer

     ServerAdmin admin@getsimpleapps.com
     ServerName getsimpleapps.com
     ServerAlias www.getsimpleapps.com
     DocumentRoot /srv/sites/getsimpleapps.com/public/
     ErrorLog /srv/sites/getsimpleapps.com/logs/error.log
     CustomLog /srv/sites/getsimpleapps.com/logs/access.log combined
</VirtualHost>

Curl from local workstation

thomas@workstation:~$ time curl -Iv https://getsimpleapps.com/
* About to connect() to getsimpleapps.com port 443 (#0)
*   Trying 50.116.58.18... connected
* Connected to getsimpleapps.com (50.116.58.18) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=Provided by New Dream Network, LLC; OU=DreamHost Basic SSL; CN=getsimpleapps.com
*    start date: 2012-02-23 00:00:00 GMT
*    expire date: 2013-02-22 23:59:59 GMT
*    subjectAltName: getsimpleapps.com matched
*    issuer: C=GB; ST=Greater Manchester; L=Salford; O=Comodo CA Limited; CN=PositiveSSL CA
*    SSL certificate verify ok.
> HEAD / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: getsimpleapps.com
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 02 Aug 2012 20:31:39 GMT
Date: Thu, 02 Aug 2012 20:31:39 GMT
< Server: Apache/2.2.22 (Ubuntu)
Server: Apache/2.2.22 (Ubuntu)
< X-Powered-By: PHP/5.3.10-1ubuntu3.2
X-Powered-By: PHP/5.3.10-1ubuntu3.2
< Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2298c7e45da25e4aaf80f7a1e36ed4a006%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A13%3A%2250.75.209.154%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A81%3A%22curl%2F7.21.4+%28universal-apple-darwin11.0%29+libcurl%2F7.21.4+OpenSSL%2F0.9.8r+zlib%2F1.2.5%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343939499%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D80bf8ae5040fc47780ccd59f1fb8b267; expires=Thu, 02-Aug-2012 22:31:39 GMT; path=/
Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2298c7e45da25e4aaf80f7a1e36ed4a006%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A13%3A%2250.75.209.154%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A81%3A%22curl%2F7.21.4+%28universal-apple-darwin11.0%29+libcurl%2F7.21.4+OpenSSL%2F0.9.8r+zlib%2F1.2.5%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343939499%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D80bf8ae5040fc47780ccd59f1fb8b267; expires=Thu, 02-Aug-2012 22:31:39 GMT; path=/
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Content-Type: text/html
Content-Type: text/html

< 
* Connection #0 to host getsimpleapps.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

real    0m29.078s
user    0m0.018s
sys 0m0.005s

Curl from linode server (via ssh)

thomas@vannevar:~$ time curl -Iv https://getsimpleapps.com/happy-ending/api/script.js?shop=holstee.myshopify.com
* About to connect() to getsimpleapps.com port 443 (#0)
*   Trying 50.116.58.18... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=Provided by New Dream Network, LLC; OU=DreamHost Basic SSL; CN=getsimpleapps.com
*    start date: 2012-02-23 00:00:00 GMT
*    expire date: 2013-02-22 23:59:59 GMT
*    subjectAltName: getsimpleapps.com matched
*    issuer: C=GB; ST=Greater Manchester; L=Salford; O=Comodo CA Limited; CN=PositiveSSL CA
*    SSL certificate verify ok.
> HEAD /happy-ending/api/script.js?shop=holstee.myshopify.com HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: getsimpleapps.com
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 02 Aug 2012 20:43:30 GMT
Date: Thu, 02 Aug 2012 20:43:30 GMT
< Server: Apache/2.2.22 (Ubuntu)
Server: Apache/2.2.22 (Ubuntu)
< X-Powered-By: PHP/5.3.10-1ubuntu3.2
X-Powered-By: PHP/5.3.10-1ubuntu3.2
< Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2204a54136cab08f9fdc5f082ebb8e739a%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2250.116.58.18%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A97%3A%22curl%2F7.22.0+%28i686-pc-linux-gnu%29+libcurl%2F7.22.0+OpenSSL%2F1.0.1+zlib%2F1.2.3.4+libidn%2F1.23+librtmp%2F2.3%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343940210%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7De7d7b8e2ca69b34c531ba7472b4b21b7; expires=Thu, 02-Aug-2012 22:43:30 GMT; path=/
Set-Cookie: ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2204a54136cab08f9fdc5f082ebb8e739a%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2250.116.58.18%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A97%3A%22curl%2F7.22.0+%28i686-pc-linux-gnu%29+libcurl%2F7.22.0+OpenSSL%2F1.0.1+zlib%2F1.2.3.4+libidn%2F1.23+librtmp%2F2.3%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1343940210%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7De7d7b8e2ca69b34c531ba7472b4b21b7; expires=Thu, 02-Aug-2012 22:43:30 GMT; path=/
< Content-Type: text/javascript
Content-Type: text/javascript
* no chunk, no close, no size. Assume close to signal end

< 
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

real    0m25.991s
user    0m0.015s
sys 0m0.022s
Community
  • 1
ThomasReggi
  • 601
  • 2
  • 9
  • 24
  • 1
    `"It turns out that this file is loading 52% slower with https (20.08s - 29.08s) that with http (380ms)."` - huh? Can you double-check your units and grammar there, please. That doesn't make much sense. – MDMarra Aug 02 '12 at 18:42
  • 1
    I think the OP meant 53 **times** slower. The HTTPS is loading really slow. –  Aug 02 '12 at 18:47
  • Maybe you just drop virtualmin on it and allow it to configure everything for you. – Andrew Smith Aug 02 '12 at 18:48
  • HTTPS takes anywhere from 20 to 29 seconds to load. HTTP takes 380 milliseconds or 0.38 seconds to load. Therefore in the time it takes one HTTPS load it could have made a HTTP connection 52 - 76 times. – ThomasReggi Aug 02 '12 at 18:48
  • Loading just the first page with "lynx -source" it 0.349s in http and 27.398s in https. I see similar slowness in Firefox. I get 5500 bytes of the same exact content either way. I think 27s is enough time to capture the process in top to see if it really using that much CPU time or is waiting for something and timing out. – Skaperen Aug 02 '12 at 18:53
  • 1
    Hmm. This is wrong. Is there anything in the Apache logs that may indicate where the slowdown is? On my server, I see it take 263ms for HTTPS and 84ms for HTTP. The very large difference you're seeing is due to something else. – cjc Aug 02 '12 at 18:54
  • 1
    Please paste your Apache configuration. – Michael Hampton Aug 02 '12 at 19:00
  • @MichaelHampton apache2.conf? – ThomasReggi Aug 02 '12 at 19:08
  • @ThomasReggi Possibly not apache2.conf. It'll be the part where you define the SSL virtual host. On Ubuntu, that may be in something like /etc/apache2/sites-enabled/default-ssl or whatever. – cjc Aug 02 '12 at 19:13
  • Updated post, check it out. – ThomasReggi Aug 02 '12 at 19:19
  • @ThomasReggi Take a look at http://serverfault.com/questions/280623/apache-extremely-slow-initial-handshake-ssl-enabled – cjc Aug 02 '12 at 19:38
  • @Skaperen how do you measure page load times with lynx. I just downloaded it for mac via homebrew and it is just getting the response body. – ThomasReggi Aug 02 '12 at 19:55
  • @ThomasReggi Use `time` and `curl`, e.g. `time curl -Iv https://getsimpleapps.com/`. We don't care about the HTML body, just the SSL handshake, anyway. – cjc Aug 02 '12 at 19:57
  • @cjc one of the times that I ran curl I got [CAfile: none](http://linux.m2osw.com/cafile_none) issue, I haven't seen it again. It usually hangs at `SSLv3, TLS handshake, Client hello (1):` I'm going to dive deeper into [this](http://serverfault.com/questions/280623/apache-extremely-slow-initial-handshake-ssl-enabled). – ThomasReggi Aug 02 '12 at 20:20
  • @cjc `cat /proc/sys/kernel/random/entropy_avail` is coming back with 168 and it should be 0? – ThomasReggi Aug 02 '12 at 20:23
  • @ThomasReggi 168 is probably OK (my own Linode is around 130). If it's zero, it's blocking. Is it zero when an HTTPS client is trying to negotiate SSL? `watch -n 0.5 cat /proc/sys/kernel/random/entropy_avail` when you run the `curl` command in a different terminal. If it's around 170, it's a different issue than the entropy pool being depleted. – cjc Aug 02 '12 at 20:27
  • @cjc I've ran curl from both the linode and locally and I haven't seen much change. Here are some entropy numbers 132, 154, 176, 134, 174 highest I've seen is 190 and lowest 120, this is without me doing anything. I've updated the question with the curl output where I get `CAfile: none` it only happens when I run curl from the linode(server). – ThomasReggi Aug 02 '12 at 20:48
  • @ThomasReggi, do a apt-get install ca-certificates – HTTP500 Aug 02 '12 at 20:50
  • Do any of your Apache config files have an `SSLRandomSeed` directive in them? – David Schwartz Aug 02 '12 at 20:58
  • The CAfile message isn't significant; it has to do with curl validating the server cert after receiving it, not with the SSL handshake being slow. – cjc Aug 02 '12 at 20:59
  • @HTTP500 `ca-certificates is already the newest version.` – ThomasReggi Aug 02 '12 at 21:00
  • Blah, you might have to start pulling out the heavy guns. Kill the running apache2 process, and start it up in debug mode (with the -X option). Figure out the process ID for the worker, and then use `strace` on it. This will spit out a ton of crap. Make a connection with curl in a different term, and see where the strace output hangs. That is where the delay is coming from. Look at http://www.arnebrodowski.de/blog/273-Entropy-drained.html for a similar problem with IMAPS, though it's possible in your case that entropy isn't the problem. – cjc Aug 02 '12 at 21:02
  • @cjc Great trick, I'm using a combination of `telnet`, `lsof`, `netstat` and `strace` but `telnet` keeps closing the output from `strace` is `23724 0.847033 accept(6, {sa_family=AF_INET6, sin6_port=htons(53668), inet_pton(AF_INET6, "::ffff:71.61.77.135", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 16`. I learned how to get the process id for the worker from [here](http://www.youtube.com/watch?v=eF-p--AH37E). – ThomasReggi Aug 02 '12 at 22:24
  • Hmm, don't use `telnet`, as it doesn't know how to do the SSL handshake. Use `curl` in a different term while you are running `strace` on the worker. That should be all you need. – cjc Aug 03 '12 at 03:03
  • How do I `curl` a specific apache worker / child process? Apache could handle it off to any of the many children. No? – ThomasReggi Aug 03 '12 at 03:12
  • @thomasreggi run Apache in debug mode. There will only be one worker. – cjc Aug 04 '12 at 00:51

4 Answers4

3

I had the same issue, with nearly identical response time differences between HTTP and HTTPS. Turns out the issue was as in the answer by @htmltiger: Apache2 was simply running out of worker processes.

This causes new requests to be queued until a worker becomes free and can process the next one [source]. I suppose the reason why this only affects HTTPS and not also HTTPS is that nearly all your traffic is over HTTP and Apache gives HTTP and HTTPS requests the same priority, taking one request from each queue in turn. So when the HTTPS queue is much longer, requests wait much longer. Indeed there are two queues, as the queue is simply the Linux TCP connection queue mechanism, and Linux provides one queue per port.

Diagnostics

If this is your problem, the following symptoms will apply:

  • The best indicator: on your server, apachectl status shows that all allowable worker processes are running. This is the case when no dots . are shwon in the process scoreboard line, indicating no "Open slot with no current process" left. The line might look like this for example:

    KKKKKKRKKKRRCWKKKCCKWKKKKCRCKKKKKKKCKCKKKKWRKKKKWRWKKKKKKCWKKWKKK

  • You see messages like this in your main Apache2 error log (/var/log/apache2/error.log, not domain specific ones):

    [mpm_prefork:error] [pid 4715] AH00161: server reached MaxRequestWorkers 
    setting, consider raising the MaxRequestWorkers setting

  • There are many processes in your Apache backlog. According to this in-depth article, you can see this from the unacked: value in ss -lti '( sport = :https )' output. Depending on the version or configuration of ss, that value might be missing though.

  • Most of the delay (say, 17 of 20 s) is shown in the Firefox Network Console, in the "Timings" tab for the initial URL requested, as "Blocking".

Solution

This assumes you use the prefork MPM server module in Apache. It's similar for the "event" and "worker" MPM modules though – details.

  1. Edit /etc/apache2/mods-enabled/mpm_prefork.conf and increase the MaxRequestWorkers setting.

  2. If you increase it beyond the default of 256, you also have to set ServerLimit to the same value to make your change effective.

  3. Apply the changes: service apache2 reload

  4. Make sure in the scoreboard output of apachectl status that the new MaxRequestWorkers setting is effective. It has to be equivalent to the length of the scoreboard line in characters.

  5. If the setting is not effective yet, search in /etc/apache2 for old configuration directives (and their even older deprecated synonyms) that could overwrite your change:

    grep -R MaxRequestWorkers /etc/apache2/*
grep -R MaxClients /etc/apache2/*

Differential Diagnoses

In case you see HTTPS being much slower than HTTP but not every single time in a series of page reloads (just on average), then you might have a variant of this fancy problem, with two Apache2 servers running on SSL port 443.

tanius
  • 590
  • 5
  • 13
0

Turns out my issue was my keys were from another server. I needed to get a new certificate and set it up with new keys.

ThomasReggi
  • 601
  • 2
  • 9
  • 24
0

I had a similar issue for a busy server but increasing MaxRequestWorkers to 400 in mpm_prefork.conf fixed it.

htmltiger
  • 11
-1

Try changing the cipher to RC4-MD5 (good balance of performance and security), i.e.:

SSLCipherSuite RC4-MD5

Cheers

HTTP500
  • 4,827
  • 4
  • 22
  • 31
  • 2
    The reported disparity between HTTP and HTTPS isn't caused by a cipher choice. It's some other misconfiguration. – cjc Aug 02 '12 at 19:15
  • @cjc I'd like to see if it makes a difference... it can't hurt to try. – HTTP500 Aug 02 '12 at 19:23
  • @HTTP500 put this in httpd.conf? What about `SSLProtocol all`? – ThomasReggi Aug 02 '12 at 19:24
  • @ThomasReggi, just put it under your SSLEngine On line. I would suggest: SSLProtocol all -SSLv2 – HTTP500 Aug 02 '12 at 19:27
  • What?! it is much faster now. I didn't restart apache2 is that okay? – ThomasReggi Aug 02 '12 at 19:32
  • @ThomasReggi, you need to restart/reload Apache – HTTP500 Aug 02 '12 at 19:35
  • It was at 7 sec before restart now its 4 sec. The 30s load hasn't came up. – ThomasReggi Aug 02 '12 at 19:38
  • @ThomasReggi, I can't explain the 7 seconds before restart, maybe cjc is on to something. Comment out the SSLCipherSuite suggestion that I made and restart and see if it goes back to the super slowness. – HTTP500 Aug 02 '12 at 19:42
  • @ThomasReggi If it was getting faster before you did that change, take a look at the entropy (with that other SF question I cited in the comments on your question). It's possible the entropy level changed because you edited the file. – cjc Aug 02 '12 at 19:42