curl bails on download in cygwin but not os x

Question

I'm trying to download a copy of a postgresql db from Amazon S3 in cygwin. But it yields zero's across the board yielding a useless file. This is my curl command:

 curl `heroku.bat pgbackups:url` -o latest.dump --verbose

which yields: * STATE: INIT => CONNECT handle 0x60002de60; line 1011 (connection #-5000) * Hostname was NOT found in DNS cache * Trying 54.231.1.232... * Adding handle: conn: 0x600069f80 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * 0x60002de60 is at send pipe head! * - Conn 0 (0x600069f80) send_pipe: 1, recv_pipe: 0 * STATE: CONNECT => WAITCONNECT handle 0x60002de60; line 1058 (connection #0) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to s3.amazonaws.com (54.231.1.232) port 443 (#0) * successfully set certificate verify locations: * CAfile: /usr/ssl/certs/ca-bundle.crt CApath: none * SSLv3, TLS handshake, Client hello (1): } [data not shown] * STATE: WAITCONNECT => PROTOCONNECT handle 0x60002de60; line 1171 (connection #0) * SSLv3, TLS handshake, Server hello (2): { [data not shown] * SSLv3, TLS handshake, CERT (11): { [data not shown] * SSLv3, TLS handshake, Server finished (14): { [data not shown] * SSLv3, TLS handshake, Client key exchange (16): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Finished (20): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): { [data not shown] * SSLv3, TLS handshake, Finished (20): { [data not shown] * SSL connection using AES128-SHA * Server certificate: Redacted * SSL certificate verify ok. * STATE: PROTOCONNECT => DO handle 0x60002de60; line 1190 (connection #0)

GET /hkpgbackups/[e-mail]/b007.dump?AWSA redacted User-Agent: curl/7.34.0 Host: s3.amazonaws.com Accept: /

 * STATE: DO => DO_DONE handle 0x60002de60; line 1263 (connection #0)
 * STATE: DO_DONE => WAITPERFORM handle 0x60002de60; line 1384 (connection #0)
 * STATE: WAITPERFORM => PERFORM handle 0x60002de60; line 1395 (connection #0)
 * HTTP 1.1 or later with persistent connection, pipelining supported
 < HTTP/1.1 400 Bad Request
 < Transfer-Encoding: chunked
 < Date: Wed, 02 Jul 2014 21:03:39 GMT
 < Connection: close
 * Server AmazonS3 is not blacklisted
 < Server: AmazonS3
 { [data not shown]
 * STATE: PERFORM => DONE handle 0x60002de60; line 1565 (connection #0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 * Closing connection 0

Running this command in OS X terminal from the same subnet yields largely the same output except I get a 200 for the http/1.1 request.

So why would I get a 200 for the Mac, but 400 for cygwin in Windows? I've tried to eliminate other potential variables, but cygwin on this machine is not getting a 200. I can tell you that the redacted url in cygwin matches the url in the heroku command output and the same in the mac as far as I can see, except for the AWS access key. Don't see why they would be different?

Why would this be different?

For reference this is my resource.

Answer 1

This should have worked, but the only way it would run is skip the evaluation of the heroku command and just copied pasted the url output AND ONLY if I enclosed the URL in single quotes.

curl -o latest.dump 'https://example.com/reallyLongAndUglyUrl'

The reason it works in OS X is because the output is very long and it doesn't get munged. But it does get munged in cygwin because I'm using the heroku.bat command to get the output. That imposes Windows constraints – that is to say, it adds a carriage return, etc to the very long line which breaks the URL. The other clue is that curl returns a 400 error which indicates a munged URL. The fix is to pipe the output to dos2unix program and curl responds as it should have.

curl -o latest.dump heroku.bat pgbackups:url | dos2unix and happiness reigns.