wget won't ignore no-follow attributes

2

I'm using the following command to download all files off of a webpage:

wget --recursive "http://example.com"

This gives me sometimes the following error:

no-follow attribute found in www.example.com. Will not follow any links on this page

According to gnu.org, I have to add -e robots=off --wait 0.25 to my command.

My final command looks like this (I don't want span-hosts):

wget --recursive -e robots=off --wait 0.5 "http://example.com"

However I am still getting the above error. What can I do to ignore those attributes?

confetti

Posted 2019-10-22T12:33:44.500

Reputation: 1 625

No answers