WGET download from https is extremly slow

1

I have performance problems with wget in case it uses https protocol. From multiple servers, most are running fine but one is 10-times slower and creating high CPU load during this time:

wget -d --inet4-only --no-check-certificate -O /dev/null https://127.0.0.1

--> 20...60s

  • In case I use the same command on the slow server but with only http:// --> 1s

  • In case I use the same command with https:// from another server to the slow server --> 1..2s

  • IPv6 is not enable, so even with --inet4-only from https:// --> 20..60s

  • The DNS-resolution is not involved due to strip the problem down even with IP-address

  • The webserver can't be the problem, because a wget from another server to the slow one is fast

  • The self-signed certificate can't be a problem, because a wget from another server to the slow one shows the same warning but is fast

All server are running under Debian Wheezy, wget version is standard "1.13.4-3+deb7u3", libgcrypt11 is 1.5.0-5+deb7u5, libgnutls26 is 2.12.20-8+deb7u5

Here is the debug output from wget -d

Setting --inet4-only (inet4only) to 1
Setting --check-certificate (checkcertificate) to 0
Setting --output-document (outputdocument) to /dev/null
DEBUG output created by Wget 1.13.4 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2016-09-27 22:08:32--  https://127.0.0.1/

******* here wget waits for some seconds *******

Connecting to 127.0.0.1:443... connected.
Created socket 5.
Releasing 0x00234188 (new refcount 0).
Deleting unused 0x00234188.

******* here wget waits for some seconds *******

WARNING: The certificate of `127.0.0.1' is not trusted.
WARNING: The certificate of `127.0.0.1' hasn't got a known issuer.
The certificate's owner does not match hostname `127.0.0.1'

---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnueabi)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found
X-Powered-By: PHP/5.4.45-0+deb7u5
Set-Cookie: PHPSESSID=g7vo4at63lg6vkmemhjns8uqt3; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: https://127.0.0.1/index.php?r=user/login
Content-type: text/html
Content-Length: 0
Date: Tue, 27 Sep 2016 20:08:42 GMT
Server: lighttpd/1.4.31

---response end---
302 Found

Stored cookie 127.0.0.1 -1 (ANY) / <session> <insecure> [expiry none] PHPSESSID g7vo4at63lg6vkmemhjns8uqt3
Registered socket 5 for persistent reuse.
Location: https://127.0.0.1/index.php?r=user/login [following]
] done.
URI content encoding = None
--2016-09-27 22:08:42--  https://127.0.0.1/index.php?r=user/login

******* here wget waits for some seconds *******

Disabling further reuse of socket 5.
Connecting to 127.0.0.1:443... connected.
Created socket 5.
Releasing 0x0022eaf0 (new refcount 0).
Deleting unused 0x0022eaf0.

******* here wget waits for some seconds *******

WARNING: The certificate of `127.0.0.1' is not trusted.
WARNING: The certificate of `127.0.0.1' hasn't got a known issuer.
The certificate's owner does not match hostname `127.0.0.1'

---request begin---
GET /index.php?r=user/login HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnueabi)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive
Cookie: PHPSESSID=g7vo4at63lg6vkmemhjns8uqt3

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
X-Powered-By: PHP/5.4.45-0+deb7u5
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: language=de; expires=Wed, 27-Sep-2017 20:08:52 GMT; path=/
Content-type: text/html
Transfer-Encoding: chunked
Date: Tue, 27 Sep 2016 20:08:52 GMT
Server: lighttpd/1.4.31

---response end---
200 OK

Stored cookie 127.0.0.1 -1 (ANY) / <permanent> <insecure> [expiry 2017-09-27 22:08:52] language de
Registered socket 5 for persistent reuse.
Length: unspecified [text/html]
Saving to: `/dev/null'

    [ <=>                                                                                                                ] 6,565       --.-K/s   in 0.008s  

2016-09-27 22:08:53 (801 KB/s) - `/dev/null' saved [6565]

Does anyone have an idea what causes this huge delay and high CPL load?

Thanks Achim

Achim

Posted 2016-09-27T20:40:31.463

Reputation: 41

Guess: it's some scheduling problem, because the slowdown only appears if both server and client run on the same computer, and communicate via localhost. What happens if you use some of the alternative algorithms for the linux kernel scheduler? – dirkt – 2016-09-27T21:24:45.300

The problem also occurs if the wget client on the slow computer accesses another computers server (that's where I originally found the problem). I have already shut down nearly all other processes, so that only wget is running - still it is terrible slow if using https... – Achim – 2016-09-29T14:55:06.417

Then I'd try to narrow down the problem by using wireshark on the fast server and observe the differences when the slow wget client access the fast server, as compared to when a fast wget client on another computer accesses the server. – dirkt – 2016-09-29T16:18:22.230

No answers