How to start using Wget?

Please, forgive me for asking this question. Usually I would try to learn thisngs myself first before bothering others, but my situation is urgent - if I don't act now and don't download all my family pictures from this website, it will be closed in about two weeks from now and I will loose all of them. So, please help me.

I have already asked a question about how to download from my website all my pictures automatically here and I was advised to use Wget. But I don't know the very basics as to how to use it. The basics are not explained on that page.

In that questioned that I asked I was even given this line:

wget -r -A .jpg,.gif,.png

but I don't know what I am supposed to do with it in order to get the Wget going and download all the pictures from my website. Please, help me somebody!

brilliant

Posted 2009-12-01T05:55:21.633

Reputation: 485

It looks like you've gotten a lot of good answers by now. In the future, consider editing your original post to include the new information that people are prompting you to provide. That way, any new visitors can find all of the info they need (e.g., that you're running Windows) all in one spot. – Miss Cellanie – 2009-12-01T23:48:13.233

You've got a lot of questions on SO and SU about wget and answer basically no other questions. I personally think if you can't install it you can't use it well. I'd recommend you try a different tool. E.G. http://blogs.techrepublic.com.com/security/?p=62

– dlamblin – 2009-12-06T03:45:57.707

Answers

Two weeks seems plenty of time to learn how to use wget and download all of your family pictures, BUT depending on the website, you may find that wget doesn't work.

This might be the case if the website creator wants to discourage bulk downloading of images. And a lot of image hosting sites do this.

I forget which hosting site it was, but I saw one which made all the image links call a JavaScript function. The actual URL of the image was passed as a parameter to the JavaScript, but it was visible in the source. wget was completely confused by this.

I have used a method which works with those image-hosting sites. Unfortunately, it requires a little 'scripting' and I'm not convinced you'll give it a try, since you learned about wget from a question on November 26, and it doesn't sound like you've tried it yet.

The first step of my method requires you to save the source of the webpage as a text file, then edit it to remove everything but the URLs which look like links to your images. (ie, starting with 'http://' and ending with .jpg, .gif, .png etc).

Then, with each URL on a separate line, insert wget and a space, before the URL, and save the edited file.

Finally you can either turn it into a shell-script or pipe it to the command shell.

As I say, this method has worked for me, and if it's a one-off thing it may be worth trying.

pavium

Posted 2009-12-01T05:55:21.633

Reputation: 5 956

sigh...This demonstrates yet again that when no OS is mentioned, the person is using Windows. Why? Because they think there is nothing else out there? Maybe... This (along with the fact that no one reads README files) is one of the main things that I find depressing in life. – marcusw – 2010-03-15T16:59:14.577

hank you for your answer, but I don't know how to install wget - I need to find out that first (please see my comment to jheddings' answer above) – brilliant – 2009-12-01T07:30:49.820

1The fact that you said 'D' disk below suggests you're using Windows. We should have asked that question right from the start. What you downloaded (wget-latest.tar.gz) looks a Linux 'tarball' not a Windows executable. I'm sure there's a Windows executable for wget – pavium – 2009-12-01T07:42:37.627

It's a little late, but there is a good port of wget for Windows. I added it to my answer. – jheddings – 2009-12-06T02:44:13.523

If you want let wget rip through your site, simply execute the line that was given, followed by your URL:

wget -r -A ".jpg,.gif,.png" http://www.mysite.com/

Note that this will traverse links on your site to find images and download them into a local directory.

EDIT To obtain wget for Windows, you can use the link from the gnuwin32 project.

jheddings

Posted 2009-12-01T05:55:21.633

Reputation: 751

Thank you for your answer, but I think I am missing the very first step in using wget - I don't know how to install it.

I've downloaded it from here: http://ftp.gnu.org/gnu/wget/ ( the version is wget-latest.tar.gz of 22-Sep-2009), saved it into one of my folders on D disk and unzipped it. I read through "READ ME", but didn't find any info there on how to install it (there was no "Install" file), so I assumed that unzipping it means installing it. But when I opened the command line and typed:

wget -r -A ".jpg,.gif,.png" <the URL of my website>

and pressed "OK", I only get an error message

– brilliant – 2009-12-01T07:28:06.283

I think I need to know how to install wget first, so I am going to ask another question. – brilliant – 2009-12-01T07:29:03.020

Wow brilliant. It's a command line tool. You type what you've been given into the command line, and on windows you probably want to have the wget.exe program in the same directory as you are in at the command prompt. If you can't use a command line tool, stop asking about wget right now and use something that doesn't require you to ask what people mean when they answer you. http://blogs.techrepublic.com.com/security/?p=62

– dlamblin – 2009-12-06T03:31:47.580

You haven't specified the OS you are using. Wget comes installed on most (all?) Linuxes by default, but not Windows or Mac OS X. You can install it for either of those though, then run it from the command line.

Rich Bradshaw

Posted 2009-12-01T05:55:21.633

Reputation: 6 324

I am using windows XP, and, now, after having read your answer, I do realize that I need to install it first, but I still don't know how. It's seems that I only know how to download it :) . I've already asked a question about installing wget. If you have time and desire, please, answer it there. Thank you. – brilliant – 2009-12-01T07:44:41.680

I can't say that I've ever used Windows for anything like this, but I reckon if you download wget you'll probably end up with a file called wget.exe. Put that in C:, then press windows+R and type in cmd and press return. In the terminal that opens, type C:, then wget.exe -r -A and your other options...

That should work! – Rich Bradshaw – 2009-12-01T10:28:16.433

Try this link btw: http://gnuwin32.sourceforge.net/packages/wget.htm

– Rich Bradshaw – 2009-12-01T10:29:25.917

I'm surprised no one mentioned Wget's official page: http://www.gnu.org/software/wget/.

Joe

Posted 2009-12-01T05:55:21.633

Reputation: 176

Throw in a --mirror and then point it a startpage.

Then you get a local mirror of the other site, hopefully with your pictures.

To install Wget

Windows:

Debian/Ubuntu

apt-get install wget

Johan

Posted 2009-12-01T05:55:21.633

Reputation: 4 827

Thank you for your answer, but I don't know how to install wget - I need to find out that first (please see my comment to jheddings' answer above) – brilliant – 2009-12-01T07:31:31.720

Don't you have ftp access to your website?

If yes then it should be easier. But of course you can use wget. Use wget --help to see command line option. Then use -m or --mirror to download your whole website.

Michał Niklas

Posted 2009-12-01T05:55:21.633

Reputation: 191

Thank you for your answer, but I don't know how to install wget - I need to find out that first (please see my comment to jheddings' answer above) – brilliant – 2009-12-01T07:30:13.657

Ooops, I am sorry I frgot to tell you that that site has blocked me from using FTP protocol because I once uploaded a file that was bigger than a file limit on that site. – brilliant – 2009-12-01T07:46:05.980

If you need wget for Windows, look for a single compiled Windows binary, it does not require any installation, you just need to have it in your PATH.

If you are confused by wget, which is a command-line tool, there are many alternatives that can help you to spider a site.

A good GUI alternative would be httrack. It is freeware, it has a GUI version, and there's a Windows version. Perhaps this will be easier for you to figure out, expecially if you have some experience with other similar tools like Teleport Pro or Offline Explorer.

geek

Posted 2009-12-01T05:55:21.633

Reputation: 7 120

Thank you, geek, for referring me to httrack. I already seem to be coming to a dead end here (http://superuser.com/questions/77279/where-on-my-computer-did-wget-download-this-image), so I do need alternatives!

– brilliant – 2009-12-01T16:36:37.380

The easiest way to have wget on your Windows system is to download the standard version of Ch Shell by SoftIntegration. Once installed, you can start the Ch shell from the shortcut it places on your desktop, and follow the commands give in the above answers.

And if for some reason wget doesn't meet your needs, Ch shell also includes curl.

Joe Internet

Posted 2009-12-01T05:55:21.633

Reputation: 5 145

Thanks for this suggestion. I'll be sure to try this one if I don't find any solution here: http://superuser.com/questions/77279/where-on-my-computer-did-wget-download-this-image

– brilliant – 2009-12-01T16:38:45.300