Check MD5 of a file header hosted on a webserver

Question

I am writing a bash script to periodically download a setup file using cURL. The problem is the following, the file is around 110 MB and always have the same filename.

I can download the file, check its hash (md5, sha1) and compare it to a previously downloaded file:

• if the hash is equal, the file is the same and has not been updated: in this case, I have downloaded it for no reason.
• if the hash is different, the file on the webserver is a new version: in this case, I use this file and update to the new version.

I would like to run this script every half-an-hour, and downloading 110MB for nothing (same file version) bothers me. Is there a way to download the first 500KB using curl in command line ?

My current workaround is to compare the filesize from the HTTP header but I still would like to have an answer on the question above and know if this is feasible. Many thanks.

curl -sI 'http://domain.tld/uri/updater/getLatest.etcenter?c=var&asi=allright' -H 'Host: domain.tld' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'DNT: 1' -H 'Connection: keep-alive' | grep 'Content-Length:'

Answer 1

Can you trust the modification dates of your file on both the client and server? If so, you can use HTTP's "conditional GET"/"If-Modified-Since" functionality with curl's -z option. No need to download anything at all. This is built-in functionality of the HTTP protocol that's been used heavily everywhere since the very beginning.

No need to download anything if the modification date of the file on the server is not newer than that of the file you downloaded last time.

For your bash script, consider using stat(1) to find the mod time of the local file, so you can feed that info into curl -z ….