Wget getting responce 403

Question

I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403.

Is there any way to check response of wget and if I get 403 to wait 5 mins and retry?

And here is my (for now) test code:

system ("wget \"http://test-link.com/403/\" -O  {$dir}/{$in_dir_counter}.xml");
$test = system ("wget \"http://test-link.com/403/\" -O  {$dir}/{$in_dir_counter}.xml");

echo "responsed - ".$test;      

both returns same.

Answer 1

How about using a simple script for that:

• Run the script once every 5 minutes unless it's running already.
• Check the age of the local file. If it's older than a specific threshold, download it again.

So if everything wents smooth, nothing happens, unless a file is outdated. If a file is outdated and failed downloading, you can retry next Iteration.

I'm not sure why you tagged this with php, but if you're actually running a php script this approach is rather easy to do (given you've got web sockets enabled):

foreach($files as $file)
    if (@filemdate($local_path + $file) + $cache_duration < time())
        @copy($remote_path + $file, local_path + $file);

Note that $remote_path can indeed be a HTTP or FTP URL. There's no need to invoke wget. The @ will prevent error messages to be printed.

To prove that this won't cause unneeded waiting:

• Assume you've got 1000 files to download, but you can only download up to 250 files per hour.
• Set cache_duration to a save time where you'll get all files, like 24h (24 * 60 * 60).
• Rerun the script above once every hour.
• The first iteration the first 250 files will be updated. The others will fail.
• The second iteration the first 250 files will be skipped (due to being recent enough) and the next 250 files will be downloaded.
• After the fourth iteration you'll have all 1000 files updated/downloaded.
• Of course you can set a shorter intervall, like 5 minutes, but this will create a lot more requests/traffic (depends on whether this is acceptable).

---

Alternative script idea:

• Try to download a file.
• If it fails, you should be able to determine that based on wget's return value/exit code. So in that case wait 5 minutes, then repeat.