2
I'm using the following command to download all files off of a webpage:
wget --recursive "http://example.com"
This gives me sometimes the following error:
no-follow attribute found in www.example.com. Will not follow any links on this page
According to gnu.org, I have to add -e robots=off --wait 0.25
to my command.
My final command looks like this (I don't want span-hosts
):
wget --recursive -e robots=off --wait 0.5 "http://example.com"
However I am still getting the above error. What can I do to ignore those attributes?