HTTP downloader with good integrity(hash..probably md5) checker built in

Question

I'm trying to download xcode v3.2 from the Apple Developer Connection website, but everytime I download the huge file, it fails to mount due to a checksum error. It's really frustrating! BitTorrent downloaders have more reliability than normal http downloads. Anyways, just wondering if there is any intelligent downloader which can download without corrupting.

I can use an md5 hash checker, but that's only useful once the whole file has been downloaded. I need a solution which can check integrity on-the-fly and re-download the corrupted bits. If there's any such provision in the HTTP.

I've searched for the file on the usual torrent sources... its not there :(

Any help is appreciated...have a good day! thanx

Answer 1

Bittorrent is all you can have at the moment. Standard HTTP downloads can't be processed as you describe -- first you'd need to checksum all the pieces of a file beforehand (like bittorrent), then you'd need to distribute a small file with those checksums to the client (like bittorrent), then you'd need software that checks those checksums as pieces come in (like bittorrent).

The best you can do with standard HTTP/FTP/etc downloads is checksum the file after the transfer is complete.

If your transfer is failing, try

• a) another download location (as in another URL to download from), or
• b) another download location (as in physically travel to another site and use their internet connection -- maybe try it from work).

Answer 2

Http doesn't tell you the checksum of the data in advance, and it certainly doesn't tell you a checksum of the parts. Without using a non-http tool, the best I could recommend is that you use curl with the resume option, I've had better luck with that, and actually did use it for large dmg files on Apple's site.

That would be using the -C - option like so:

curl -O -C - http://developer.apple.com/download/something.dmg