How do I properly set wget to download only new files?

42

8

Let's say there's an url, let's call it http://www.some-url.com/folder/

This location has directory listing enabled, therefore I can do this:

wget -r -np http://www.some-url.com/folder/

To download all its contents with all the files and subfolders and their files.

Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?

Richard Rodriguez

Posted 2011-05-14T16:54:11.517

Reputation: 1 676

Answers

55

did you read this ? http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html

"Several days later, you would like Wget to check if the remote file has changed, and download it if it has."

wget -N http://www.some-url.com/folder/

Try this.

user69274

Posted 2011-05-14T16:54:11.517

Reputation: 668

6But -N is useless if retrieved file has not Last-Modified header. – unibasil – 2017-12-29T04:27:58.223

18

wget -c 

also continues partial files caused when the download has issues.....
Better yet,

 wget -c -N 

seems to do both at the same time.

Henry Crutcher

Posted 2011-05-14T16:54:11.517

Reputation: 281

I found with my version 1.11.4 Red Hat modified, these flags are not compatible and -c overrides -N. – Matt Williamson – 2016-09-21T14:25:31.477

-c option of wget helped mw the sync partially downloaded file from url. Its really a good option to use if faced any network problems while downloading. – mchawre – 2020-02-15T16:43:29.493