How can I download a file the moment it becomes available on linux?

Question

How can I set up something on a Linux machine so that the moment a file on a web server returns something other than a 404 error it will download it. I would like it to stop checking the moment the file is started to download successfully, but still retry if it fails.

I have tried using a cron job with wget, but I can't seem to figure out how to make it not create an empty file on a 404 error.

Is this possible and how?

Answer 1

You can either check (with test -s) that the file has contents, or simply use the return value of wget. Download to a temporary file, and only if the test passes, then copy to the real output:

$ wget -q -O /tmp/a http://localhost/nonexistent && mv -v /tmp/a /tmp/b
$ wget -q -O /tmp/a http://localhost && mv -v /tmp/a /tmp/b
‘/tmp/a’ -> ‘/tmp/b’
$

Answer 2

You can write a bash script that will wget the url and then will check the output. If the output contains lets say "404", delete it. Else you got your file, and you can remove the script from cron (from within the script). You can also use a scripting language to solve this.