2
I'm looking for a shell command that robustly ensures a file is completely downloaded, but that avoids redownloading anything unnecessarily. Here's pseudocode of what I'm hoping for:
If file doesn't exist, download it.
If file exists:
use HTTP HEAD to get timestamp and size of remote file.
if remote timestamp is newer, delete local file and download remote file
if timestamps are the same:
if remote size is greater than local size:
resume download
if remote size is equal to local size:
do nothing
if remote size is less than local size:
do nothing but issue a warning because this is weird
wget
is able to resume interrupted transfers using the -c
option, but I have to track somewhere that it got interrupted so I can know to pass in that option, and it requires running the command over again.
wget -N
makes sure to avoid downloading a file if the timestamp of the remote file is not newer than the local one. But it doesn't know if transfers were interrupted and thus won't do anything when called again on an interrupted transfer.
curl -C -
will download a file if not present and resume it if only partially downloaded. But if a completely downloaded file is present, it gives me errors about the server not supporting byte ranges.
I suppose I could write something to implement my pseudocode myself, but this feels like it would be a common enough desire that I ask, is there an existing way of doing this?
1This seems to work nicely. Not sure if it respects the timestamps as I outlined, but it's basically what I need. – Josh Hansen – 2017-02-10T22:05:34.847