Wget wait functionality not working as expected

3

Background:
I am using wget (version 1.14-15.el7) on RedHat Linux (version 7.3 (Maipo)), in a Bash (version 4.2.46(1)) terminal.

Unfortunately, I am limited to this particular RHEL image, so I can't upgrade to a newer version of wget.

Goal:
I am trying to configure wget so that, if a download attempt fails, it does the following:
1) Retries the download 4 more times (total of 5 attempts)
2) Waits a FIXED amount of time (30 seconds) between download attempts ('retries')

For context, here is a relevant snippet of the wget manual:

   -w seconds
   --wait=seconds
       Wait the specified number of seconds between the retrievals.  Use of this option is
       recommended, as it lightens the server load by making the requests less frequent.
       Instead of in seconds, the time can be specified in minutes using the "m" suffix, in
       hours using "h" suffix, or in days using "d" suffix.

       Specifying a large value for this option is useful if the network or the destination
       host is down, so that Wget can wait long enough to reasonably expect the network error
       to be fixed before the retry.  The waiting interval specified by this function is
       influenced by "--random-wait", which see.

   --waitretry=seconds
       If you don't want Wget to wait between every retrieval, but only between retries of
       failed downloads, you can use this option.  Wget will use linear backoff, waiting 1
       second after the first failure on a given file, then waiting 2 seconds after the second
       failure on that file, up to the maximum number of seconds you specify.

       By default, Wget will assume a value of 10 seconds.

To be clear, I am using the --wait flag, NOT the --waitretry flag.

Process:

First, I export/set a wrong http_proxy and https_proxy, to ensure any download attempt will time out.

I run the following command: wget --tries=5 --wait=30 <url> -O <output_filename>

At this point, the --wait functionality does NOT work as expected. Specifically, it does NOT wait 30s after each download attempt.

Instead:
1) After the first attempt, it waits 1s.
2) After the 2nd attempt, it waits 2s.
3) After the 3rd attempt, it waits 3s.
and so on ...

In other words, despite using the --wait flag (which should result in a fixed waiting time between download attempts), wget seems to be performing a 'linear backoff' as described in the --waitretry flag section.

PROBLEM:
I want the functionality of the --wait flag, NOT the --waitretry flag.

Unfortunately, the --wait flag seems to be acting like the --waitretry flag -- is there any way to fix this apparent bug in wget, so that using the --wait flag results in the expected fixed waiting time between download attempts?

Seth

Posted 2017-08-18T00:25:55.780

Reputation: 131

1Have you tried setting wait before tries? – Anaksunaman – 2017-08-18T02:11:11.497

Thanks, but changing the order of the flags made no difference. – Seth – 2017-08-18T02:46:38.720

1--wait is only for waiting between downloading different files, it's not related at all to retrying failed downloads. – Barmar – 2017-08-18T19:29:20.563

1@Barmar I'm not sure. The fragment: "this option is useful if the network or the destination host is down, so that Wget can wait long enough" is related to retrying failed downloads, isn't it? – Kamil Maciorowski – 2017-08-18T19:42:18.560

1

Can you use a shell loop with a wait command and set wget to try the download only once? Are you downloading multiple files / recursively? Any why are you needing this fixed timing? Just asking to prevent a xy-Problem...

– mpy – 2017-08-22T19:30:46.457

@mpy: Thanks, that's actually what I ended up doing. However, I was hoping for a solution that relied only on native wget functionality, as opposed to me having to write a custom shell loop that invokes wget. I am only downloading 1 file, but the site I'm downloading it from might occasionally be down. The fixed timing is just to ensure that: If a download fails, I retry it, but I also don't spend too long fruitlessly retrying over and over again -- the specific numbers are basically arbitrary. – Seth – 2017-08-23T20:26:09.403

Answers

1

Whenever I find a linux utility isn't acting as described in the man pages, I check the BSD man pages. They are often filled with additional valuable info.

I found this at the end of the --waitretry section: Note that this option is turned on by default in the global wgetrc file.

Potentially you have a global or user wgetrc value defining waitretry.

I'm not 100% sure of the location of the file on your RHEL version. Possible options: ~/.wgetrc /etc/wgetrc /usr/local/etc/wgetrc https://www.gnu.org/software/wget/manual/wget.html#Startup-File

CoreyJJohnson

Posted 2017-08-18T00:25:55.780

Reputation: 126