18

Here is a link given on curl's official website:

(prefix omitted) bintray.com/artifact/download/vszakats/generic/curl-7.46.0-win64-mingw.7z

When I downloaded it with prefixes http:// and https:// I got two different files.

My question is why is this site serving two different files -- one over HTTP and one over HTTPS? The SHA256 hashes do not match.

Here is a diff of the final URLs after redirect. The URL on the left is the one that HTTP redirects to and the URL on the right is the one HTTPS redirects to:

Diff of URLs

Update:

The two files were not downloaded concurrently, but they had the same version number, so I assumed they should be the same. This is not the case. The author told me some revisions do not get a new version number. The scan results for the files served from the site on different dates (12/05/15 was the scan date for the download over HTTP and 12/28/15 was the scan date for the download over HTTPS) led to confusion because the version number did not change but the SHA256 hash did.

solarflare97
  • 191
  • 1
  • 7
  • 2
    Another possibility is that you have HTTPS-only cookies set in your browser, and the website is sending different content in response to that input. – AKHolland Dec 28 '15 at 16:18
  • 3
    A third possibility is that you are being MITMed and the HTTP version has malware inserted. – user253751 Dec 29 '15 at 00:50
  • @immibis - Yes, HTTP is subject to MITM. While additional research revealed this was not the case in this instance, I think that is a very important thing to be aware of. – solarflare97 Dec 29 '15 at 05:19

4 Answers4

24

The simple answer is: because it wants to! The web server can serve whatever it likes, either by configuration or coincidence.

Right now, I get the same 75916c7b file over both HTTP and HTTPS and cannot confirm your theory that the web server is serving different content for HTTPS versus HTTP. However, if you managed to access the site near the time the file was updated, it could very well be that different servers were serving the different protocols and the file update had not yet propagated to the server that served you the old file.

Remember that one URL can be served by any number of servers - the fact that you get one file from a URL does not exclude there being 20 copies of this file on 20 servers/caches, some of which may be out of date.

This a certainty in this case, as the website appears to be using Cloudfront which is a Content Delivery Network - a piece of infrastructure explicitly designed for caching files on many distributed servers for delivery at global scale.

Sander
  • 352
  • 1
  • 6
  • One installer for one product with one version number should never have a different hash, though. It does seem a bit fishy. – Lightness Races in Orbit Dec 28 '15 at 14:17
  • Something that doesn't quite add up is that `curl-7.46.0-win64-mingw.7z` should either be served or completely unavailable, no? If the file was updated then the different file would be named `curl-7.46.1-win64-mingw.7z`, right? – MonkeyZeus Dec 28 '15 at 14:23
  • @MonkeyZeus - [this answer](http://security.stackexchange.com/a/109130/10885) by curl author states that the bits changing without the version being revved isn't abnormal. Seems weird to me too, but s/he would know. – Neil Smithline Dec 29 '15 at 02:01
  • It ultimately boiled down to the fact that the same link to a file with the same version number was serving different versions at different dates. So, the issue was actually version control instead of protocol. While the premise that led to my question was flawed, I think the question still has merit because it is a reminder that HTTP can be subject to MITM. – solarflare97 Dec 29 '15 at 05:13
  • Here is the google cache with the same file name and a different checksum: [cache](http://webcache.googleusercontent.com/search?q=cache:sAh2mb9Z108J:https://bintray.com/vszakats/generic/curl/+&cd=1&hl=en&ct=clnk&gl=us). Here is the current site, same file name but different checksum: [current](https://bintray.com/vszakats/generic/curl/). – solarflare97 Dec 29 '15 at 05:37
14

Author of above referenced files here.

The checksum of said files at the current state of things is normal to change when the build scripts are modified and a new build automatically triggered and uploaded. In case of curl 7.46.0, the build scripts (and consequently the downloadable packages) changed three times as of this writing:

  • HTTP/2 support was enabled, by building and static linking the nghttp2 dependency: commit, log
  • nghttp2 dependency was updated from version 1.5.0 to 1.6.0: commit, log
  • a libcurl.dll build bug was identified and fixed, that was caused by a missing nghttp2 build option: commit, log
  • underlying MinGW C compiler was updated to 5.3.0, plus multiple internal fixes and updates: commit, log
  • VirusTotal false positive was fixed by dropping an optional .vbs file that was previously copied from the original curl source package: commit, log

Any commits here (made to the master branch), will trigger a new build:

  • except when explicitly excluded using commit text: [ci skip]
  • the build process is designed to be reproducible/deterministic, so the checksum will only change if the underlying source code, or the compiler options are altered. (A repeated build with the same options won't change it.)

As for downloading the binaries via different protocols, HTTP vs. HTTPS should result in the exact same binary content (if the downloads are made at the same point in time) — though HTTPS is highly recommended.

As for potential malware, see my thoughts on GitHub: https://github.com/bagder/curl/issues/583#issuecomment-167520488

In short: They are almost certainly false positives, unlikely to be influenced by the transfer protocol, more likely influenced by scanner engine problems. The complete build process is public and auditable as well as all the source code that gets built into it.)

UPDATE [2015-12-28]: As a general answer to the HTTP vs. HTTPS issue: The latter, secure protocol makes certain guarantees that the content comes from the right party and that it's not tampered on its way to the other end of the wire. For these reasons, it can be trusted more and hence my recommending it in the original answer. An even better assurance is to verify SHA256 hashes of the downloaded content against the ones generated on the build server itself (visible at the end of the build logs linked above). An even stronger assurance is to have the content digitally signed (f.e. using PGP or minisign), and that signature verified by the receiver. (My curl downloads don't feature a digital signature at this time.)

UPDATE [2016-01-06]: Updated the list of package updates. Fixing the VirusTotal false positive was one of them.

vsz
  • 186
  • 6
  • Thank you very much for your input! Also I changed the question to be more general about the HTTP vs HTTPS issue and removed discussion of malware because I agree 100% they were probably false positives. – solarflare97 Dec 28 '15 at 11:58
  • 1
    Thank you! Noted and I've added a short, general answer to the HTTP vs. HTTPS issue, which I missed from the initial response. – vsz Dec 28 '15 at 12:09
3

Look at the timestamps on the scans, the "Clean" version was weeks ago. When you click the link for a newer scan date you see both http and https are infected.

Basically there is no difference in the files, just the scans were run at two different times (pre-infection and post-infection)

Steven Carlson
  • 131
  • 5
2

Serving different files with HTTP and HTTPS is actually routine. For example, suppose you are a bank. If somebody gives the url http://www.bank.com, the bank doesn't want to use HTTP and it doesn't want to hassle you, so it just issues an answer with a 301 or 302 redirect to https://www.bank.com. That is as transparent as possible.

user95495
  • 21
  • 1
  • 2
    This seems to me to be the answer to the question the OP is asking; although the solution to his conundrum is that his observation is based on a false premise. – Floris Dec 28 '15 at 23:31
  • 1
    @Floris - Indeed, my solution was based on a false premise. My assumption was two files with the same version number should always have the same hash. Any change to the file should have a new version number. That turned out not to be the case. The HTTP download was not concurrent with the HTTPS download, so I was hashing two different versions with the same version number. It led to confusion. – solarflare97 Dec 29 '15 at 05:24