cURL: how to suppress strange output when redirecting?

Question

I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell.

But when I redirect stdout like this:

curl -v http://somehost/somepage > /dev/null

Some sort of results table appears in the middle of the output to stderr:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Followed by this near the end:

{ [data not shown]
118   592    0   592    0     0  15714      0 --:--:-- --:--:-- --:--:-- 25739

Which makes the response headers less readable.

I don't see this text when not redirecting.

---

Another way to see the effects:

Table doesn't appear:

curl -v http://somehost/somepage 2>&1

Table appears:

curl -v http://somehost/somepage 2>&1 | cat

---

1) How come this shows up only with certain types of redirects?

2) What's the neatest way to suppress it?

Thank you

Answer 1

Try this:

curl -vs -o /dev/null http://somehost/somepage 2>&1

That will suppress the progress meter, send stdout to /dev/null and redirect stderr (the -v output) to stdout.

Answer 2

curl --fail --silent --show-error http://www.example.com/ > /dev/null

This will suppress the status dialog, but will otherwise output errors to STDERR.

user@host:~# curl http://www.yahoo.com > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  254k    0  254k    0     0   403k      0 --:--:-- --:--:-- --:--:--  424k

The above outputs the status table when redirecting.

user@host:~# curl --fail --silent --show-error http://www.yahoo.com > /dev/null

The above suppresses the status table when redirecting, but errors will still go to STDERR.

user@host:~# curl --fail --silent --show-error http://www.errorexample.com > /dev/null
curl: (6) Couldn't resolve host 'www.errorexample.com'

The above is an example of an error to STDERR.

user@host:~# curl -v --fail --silent --show-error http://www.errorexample.com > ~/output.txt 2>&1
user@host:~# cat ~/output.txt 
* getaddrinfo(3) failed for www.errorexample.com:80
* Couldn't resolve host 'www.errorexample.com'
* Closing connection #0
curl: (6) Couldn't resolve host 'www.errorexample.com'

Just add 2>&1 to the end to redirect STDERR to STDOUT (in this case, to a file).

Answer 3

According to man curl:

-s, --silent : Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.

Example usage:

curl -s 'http://www.google.com'

or if you want to capture the HTTP-BODY into a variable in bash

BODY=$( curl -s 'http://www.google.com' )
echo $BODY

You can use -s or --silent interchangeably.

Answer 4

With reference to question 1 (how cURL knows to only display the table when output is redirected), I didn't realise a program could tell its outputs were being directed, but it seems on POSIX systems there is a function isatty which reports whether or not a file descriptor refers to a terminal.

Answer 5

1) How come this shows up only with certain types of redirects?

from the curl man page

If you want a progress meter for HTTP POST or PUT requests, you need to redirect the response output to a file, using shell redirect (>), -o [file] or similar.

curl must use isatty to determine the redirect and prints the progress meter when redirected to a file or shell pipe.

2) What's the neatest way to suppress it?

from the curl man page

-s, --silent

Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.

Answer 6

To put real error messages somwhere, you should write strerr into a log file. Something like that:

curl  "http://domain.name/process" --stderr /var/log/curl_err.log > /dev/null

Answer 7

Being behind a proxy I use a command like this.

date -s "$(curl --proxy http://PROXY:8080 -s http://google.com --head -s |grep Date|sed 's/Date: //g')"