WGET downloading extremely slow

3

I am using the following command

wget -r -np -k -p http://www.site.com/dir/

To download a small archive of files(there is a couple hundred files so I don't want to click download on each one and put them in the respective folders).

I am getting speeds of around 17kb/s this can't be possible. While I watch wget download the file, I open up google chrome and download the same 10 megabyte file INSTANTLY and watch as wget is still only around 50%...

I went to sleep and left my computer on and it is still downloading and not even a quarter done... any explanation as to why wget downloads 5000 times slower than my internet speed? How can I speed this up? Is there an alternate program to accomplish this? Thank you for your assistance.

CodeCamper

Posted 2014-07-16T18:05:28.450

Reputation: 149

Maybe you have something in your wgetrc file that limits the download speed? – darnir – 2014-07-16T18:55:05.680

@darnir Where can I find my wgetrc file? – CodeCamper – 2014-07-16T19:04:53.353

I'm assuming you're on a *nix machine. Check ~/.wgetrc if it exists or /etc/wgetrc. You can get the precise location of the wgetrc files your wget reads by executing: wget -v | grep -A1 Wgetrc – darnir – 2014-07-16T19:10:56.103

Also, it might be useful if you could share the debug output of wget. Run wget -r -np -k -p --debug http://www.site.com/dir/. You can hit Ctrl-C after a couple of files have been downloaded and share the output. It might give some insight. – darnir – 2014-07-16T19:12:46.183

Answers

1

Try adding the option:

-4

Also, try to run a speedtest:

wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
chmod a+rx speedtest_cli.py
sudo mv speedtest_cli.py /usr/local/bin/speedtest-cli
sudo chown root:root /usr/local/bin/speedtest-cli
speedtest-cli

This will output, among others, your download and upload speed. By the way, are you using wireless or ethernet?

You can also try an alternative to wget, for example HTTrack, aria2, cURL, etc.

William Edwards

Posted 2014-07-16T18:05:28.450

Reputation: 303

Right now I am using Ethernet, the connection is solid. Tried your suggestion not going any faster I don't think. Thanks for the alternatives by the way! Definitely checking them all out! – CodeCamper – 2014-07-16T19:00:49.243

1

@CodeCamper No problem :) Check out http://alternativeto.net/software/wget/ for more alternatives. http://alternativeto.net is definitely a website that you will find useful some day :)

– William Edwards – 2014-07-16T19:05:58.560

I don't think --no-dns-cache is a valid solution. If anything it will increase the latency of resolving the hostname a couple hundred times extra! – darnir – 2014-07-16T19:08:02.087

@darnir I think in most cases this is indeed not a good solution. I edited my answer, thanks. – William Edwards – 2014-07-16T19:10:01.150

@WilliamDavidEdwards LOL true story, I have had WGET running for 12 hours!!! I just downloaded HTTRack and I have already surpassed my progress... Something is SERIOUSLY wrong with WGET... any suggestion to configure HTTrack the same way I had WGET, I added pdf to list of file extensions but I am afraid this might crawl up the site and download things I don't want, I only want it to go down directories, also liked how WGET localized the index pages for me. I am watching WGET go 8% and 5 minutes later 9%... HTTrack just finished the whole folder... and WGET is downloading 1 file! – CodeCamper – 2014-07-16T19:13:35.707

@CodeCamper Wow, that's indeed strange... Does this happen on another device on the same network too? In that case, there's probably something wrong with your network. Good luck! – William Edwards – 2014-07-16T19:15:01.110

I'm honestly surprised at how Wget is behaving here. This isn't expected behaviour. However, I still do not understand where exactly the error resides. – darnir – 2014-07-16T19:22:32.577

@darnir I agree. I have actually never had a problem with wget being slow... It always works perfectly. – William Edwards – 2014-07-16T19:23:12.357

@WilliamDavidEdwards I did have to change one setting in HTTrack there was an option to limit bandwidth around 25kb so I moved that infinitely higher. Maybe WGET has a similar bottleneck setting? – CodeCamper – 2014-07-16T19:28:59.057

@CodeCamper That could be possible, to be honest I have no idea. – William Edwards – 2014-07-16T19:30:36.420

@WilliamDavidEdwards I think the multiple download threads is also making a difference although the download speed in WGET is the direct problem. Kind of strange, actually I think HTTrack is still going slower than my base internet speed as well. But the more settings I adjust the higher the bandwidth is going. I wish there was a recursive download function right in Google Chrome! – CodeCamper – 2014-07-16T19:36:43.643

@CodeCamper You might want to take a look at ScrapBook – William Edwards – 2014-07-16T19:38:52.940

@WilliamDavidEdwards Will I be able to download a bunch of PDFs with it? Also is this it? Because I can't seem to find a download link... https://code.google.com/p/chrome-scrapbook/

– CodeCamper – 2014-07-16T19:46:37.467

@CodeCamper I have no idea, I never used it before. The ScrapBook I found was a Firefox extension but the one you found should work as well :) – William Edwards – 2014-07-16T19:47:50.303

Asked: 2014-07-16T18:05:28.450

Viewed: 9 863 times

Active: 2016-04-02T15:40:51.907