91

I was watching the funny server type from http://www.reddit.com with curl -I http://www.reddit.com when I guessed that curl -X HEAD http://www.reddit.com would do the same. But, in fact, it doesn't.

I'm curious about why.

This is what I observe running the two commands:

  • curl -I: works as expected, outputs the header and exists.

  • curl -X HEAD: does not show anything and seems to wait for user input.

But, sniffing with tshark I see the second command actually sends the same HTML query and receives the correct answer, but it does not show it and it doesn't close the connection.

curl -I

0.000000 333.33.33.33 -> 213.248.111.106 TCP 59675 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=47267342 TSER=0 WS=6
0.045392 213.248.111.106 -> 333.33.33.33 TCP http > 59675 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=2552532839 TSER=47267342 WS=1
0.045441 333.33.33.33 -> 213.248.111.106 TCP 59675 > http [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=47267353 TSER=2552532839
0.045623 333.33.33.33 -> 213.248.111.106 HTTP HEAD / HTTP/1.1
0.091665 213.248.111.106 -> 333.33.33.33 TCP http > 59675 [ACK] Seq=1 Ack=155 Win=6432 Len=0 TSV=2552532886 TSER=47267353
0.861782 213.248.111.106 -> 333.33.33.33 HTTP HTTP/1.1 200 OK
0.861830 333.33.33.33 -> 213.248.111.106 TCP 59675 > http [ACK] Seq=155 Ack=321 Win=6912 Len=0 TSV=47267557 TSER=2552533656
0.862127 333.33.33.33 -> 213.248.111.106 TCP 59675 > http [FIN, ACK] Seq=155 Ack=321 Win=6912 Len=0 TSV=47267557 TSER=2552533656
0.910810 213.248.111.106 -> 333.33.33.33 TCP http > 59675 [FIN, ACK] Seq=321 Ack=156 Win=6432 Len=0 TSV=2552533705 TSER=47267557
0.910880 333.33.33.33 -> 213.248.111.106 TCP 59675 > http [ACK] Seq=156 Ack=322 Win=6912 Len=0 TSV=47267570 TSER=2552533705

curl -X HEAD

34.106389 333.33.33.33 -> 213.248.111.90 TCP 51690 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=47275868 TSER=0 WS=6
34.149507 213.248.111.90 -> 333.33.33.33 TCP http > 51690 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=3920268348 TSER=47275868 WS=1
34.149560 333.33.33.33 -> 213.248.111.90 TCP 51690 > http [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=47275879 TSER=3920268348
34.149646 333.33.33.33 -> 213.248.111.90 HTTP HEAD / HTTP/1.1
34.191484 213.248.111.90 -> 333.33.33.33 TCP http > 51690 [ACK] Seq=1 Ack=155 Win=6432 Len=0 TSV=3920268390 TSER=47275879
34.192657 213.248.111.90 -> 333.33.33.33 TCP [TCP Dup ACK 15#1] http > 51690 [ACK] Seq=1 Ack=155 Win=6432 Len=0 TSV=3920268390 TSER=47275879
34.823399 213.248.111.90 -> 333.33.33.33 HTTP HTTP/1.1 200 OK
34.823453 333.33.33.33 -> 213.248.111.90 TCP 51690 > http [ACK] Seq=155 Ack=321 Win=6912 Len=0 TSV=47276048 TSER=3920269022

Any idea about why this difference in behaviour?

chmeee
  • 7,270
  • 3
  • 29
  • 43

4 Answers4

82

It seems the difference has to do with the Content-Length header and how it is treated by both commands.

But before going into that, curl -X HEAD does not give any output because, by default, curl does not print headers if switch -i is not provided (not needed on -I though).

In any case, curl -I is the proper way to fetch the headers. It just ask for the header and close the connection.

On the other hand curl -X HEAD -i will wait for the transmission of the number of bytes stated by Content-Length. In the case no Content-Length is not specified, I guess it will wait for some data or for that particular header.

Some examples that shows this behaviour:

$ curl -X HEAD -i http://www.elpais.es
HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: http://www.elpais.com/
Date: Wed, 12 May 2010 06:35:57 GMT
Connection: keep-alive

Because Content-Length is 0, in this case both commands behave the same. And the connection is closed afterwards.

$ curl -X HEAD -i http://slashdot.org
HTTP/1.1 200 OK
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001296
X-Bender: Since I love you all so much, I'd like to give everyone hugs.
X-XRDS-Location: http://slashdot.org/slashdot.xrds
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
Content-Length: 115224
Date: Wed, 12 May 2010 06:37:20 GMT
X-Varnish: 1649060825 1649060810
Age: 1
Connection: keep-alive

curl: (18) transfer closed with 115224 bytes remaining to read

In this case, there seems to be a timeout (probably by Varnish), so curl protests that the connection was closed before having received the Content-Length number of bytes.

By the way, look at the funny X-Bender (shown in the example) and X-Fry (try it for yourself) headers :).

chmeee
  • 7,270
  • 3
  • 29
  • 43
  • 2
    In case anyone else comes looking for this: the option to set in PHP's curl library is `CURLOPT_NOBODY`. – Matthew Jul 13 '12 at 14:39
13

I think this is a bug in curl. If I specify a method with -X, curl should handle the response according to the RFC. Unfortunately, the maintainer of curl does not agree. Someone filed a bug and even submitted a patch:

http://sourceforge.net/tracker/?func=detail&atid=100976&aid=1810273&group_id=976

but the curl maintainer rejected it. Apparently a broken "-X HEAD" option is "working as designed".

--Jamshid

jamshid
  • 241
  • 2
  • 5
  • 7
    To be fair, I can follow the logic of the ticket response: `--head` does provide us with a valid implementation of a HEAD request, and `-X ` simply overrides the HTTP method in the request. – Hank Feb 20 '14 at 11:07
  • 4
    yes, this was actually what I needed. I have a buggy server that serves up content when given a HEAD request. `-X HEAD` was the only way I could test it when trying to get the server to adhere to the RFC – Hashbrown Aug 21 '17 at 08:13
12

From the docs:

-X, --request

(HTTP) Specifies a custom request method to use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET). Read the HTTP 1.1 specification for details and explanations. Common additional HTTP requests include PUT and DELETE, but related technologies like WebDAV offers PROPFIND, COPY, MOVE and more.

Normally you don't need this option. All sorts of GET, HEAD, POST and PUT requests are rather invoked by using dedicated command line options.

This option only changes the actual word used in the HTTP request, it does not alter the way curl behaves. So for example if you want to make a proper HEAD request, using -X HEAD will not suffice. You need to use the -I, --head option.

In other words, -X is for methods other than GET, HEAD, POST and PUT. For HEAD use -I.

Community
  • 1
x-yuri
  • 1,845
  • 1
  • 22
  • 27
0

I meet same issue when writing cpp code on curl 7.34,

curl_easy_setopt(curl_handle, CURLOPT_CUSTOMREQUEST, "HEAD");

will hang there a long time, seems it is waiting for body transfer until a timeout happens. after adding a new line, this issue is resolved.

curl_easy_setopt(curl_handle, CURLOPT_NOBODY, 1L );

from the doc

do the download request without getting the body

this line would force curl not to wait.

dasons
  • 101