How accurate is CURL's 'current rate' in the progress meter?

I looked at this question a few days ago and I must say that I have wondered the same at times. By the way, it isn't nearly as accurate when you analyse it at packet level. To top that, even the values within the status output do contradict. See below for example:(check the values of Current Speed in Bytes/sec and relate with Average Download Speed and Received Data in Bytes)

% Total     % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
0     0     0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0     0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2 10.0M     2  234k    0     0   161k      0  0:01:03  0:00:01  0:01:02  161k
7 10.0M     7  769k    0     0   314k      0  0:00:32  0:00:02  0:00:30  314k
10 10.0M   10 1102k    0     0   319k      0  0:00:32  0:00:03  0:00:29  319k
16 10.0M   16 1671k    0     0   368k      0  0:00:27  0:00:04  0:00:23  368k
23 10.0M   23 2371k    0     0   435k      0  0:00:23  0:00:05  0:00:18  480k
28 10.0M   28 2905k    0     0   450k      0  0:00:22  0:00:06  0:00:16  534k
31 10.0M   31 3269k    0     0   434k      0  0:00:23  0:00:07  0:00:16  492k
38 10.0M   38 3973k    0     0   470k      0  0:00:21  0:00:08  0:00:13  574k
43 10.0M   43 4476k    0     0   473k      0  0:00:21  0:00:09  0:00:12  569k
47 10.0M   47 4814k    0     0   459k      0  0:00:22  0:00:10  0:00:12  486k
54 10.0M   54 5571k    0     0   486k      0  0:00:21  0:00:11  0:00:10  533k
59 10.0M   59 6089k    0     0   474k      0  0:00:21  0:00:12  0:00:09  532k
63 10.0M   63 6522k    0     0   478k      0  0:00:21  0:00:13  0:00:08  492k
68 10.0M   68 6992k    0     0   481k      0  0:00:21  0:00:14  0:00:07  496k
74 10.0M   74 7599k    0     0   485k      0  0:00:21  0:00:15  0:00:06  536k
80 10.0M   80 8255k    0     0   489k      0  0:00:20  0:00:16  0:00:04  494k
84 10.0M   84 8648k    0     0   491k      0  0:00:20  0:00:17  0:00:03  535k
89 10.0M   89 9202k    0     0   493k      0  0:00:20  0:00:18  0:00:02  534k
95 10.0M   95 9745k    0     0   495k      0  0:00:20  0:00:19  0:00:01  536k
100 10.0M 100 10.0M    0     0   507k      0  0:00:20  0:00:20 --:--:--  584k

Now look at the sniffer trace analytical summary of the first and second second of this stream against the entire stream in pictures below: (Difference between these values and curl's status output is indeed remarkable)

First second:

First second data rate

Second second:

Second-second data rate

I believe this does prove the point that what curl reports isn't what you'd call accurate but does serve the purpose more or less. The average values that it reports are averaged over past 5 seconds from fifth second onwards and even the average calculation isn't precise.

Another point worth noticing is how curl always shows upload speed as 0 whereas sniffer trace shows the average upload speed to be around 22558 Bytes/sec for this 20 second 11MB transaction.

Upload speed for e

Hope that helps!

Donbhupi

Posted 2015-03-02T05:01:14.870

Reputation: 126

Thank you for the excellent analysis! The takeway would then be that the CURL progress meter is best limited to comparisons between downloads, or as what I am currently using it for, to generate "download bitrate shape" graphs. As with any user-level calculation, the bitrate is a function not just of the network, but also the delivery to the application, as well as the quirks of the application's measurement methodology. – Mark Gerolimatos – 2015-03-07T23:53:24.010

I believe one reason for the different results is that curl measured the application/TCP payload traffic, while wireshark measured raw traffic including Ethernet+IP+TCP header. The upload traffic is just due to pure ACKs. – Trần Việt Hoàng – 2019-04-18T09:25:12.747

What's accurate to you? The rate over the last second, or the rate from the whole download? Maybe somewhere in between? – Brad – 2015-03-02T05:25:39.987

The claim is that the bitrate is over the last second. It could be that it's just random stuff that is printed out to keep the downloader happy that something is happening. The best possible result would be the actual-bits-from-the-last-60-seconds/60-seconds – Mark Gerolimatos – 2015-03-03T01:16:08.203

It's accurate. You can test this yourself with a packet sniffer. – Brad – 2015-03-03T01:17:21.950

Thanks! That's EXACTLY what I needed to know! If you state it in the form of an answer, I'll give you credit :-) – Mark Gerolimatos – 2015-03-03T01:18:47.133

Maybe, but what I can't answer is what is the time period being quantized. And, I see now that you edited your comment asking about the data from the last 60 seconds. I think your first comment is correct, that it's probably the average bitrate over the last second as I see it swing up and down wildly sometimes. But, you would have to inspect the source to be sure. In any case, I promise you that it isn't just some made up number. It's a count of data transferred over time... just what time period I'm not sure. – Brad – 2015-03-03T01:20:26.607

Thank you. I don't have the bandwidth (excuse the pun) for verification, and frankly do not need super accuracy, just want to make sure that the RELATIVE numbers are accurate. When you graph this data, it often becomes immediately apparent as to the cause of the download slowness (ISP limiting, tail drop, good ol' congestion or CDN funny business). And that is why accuracy was somewhat important, so that statements made about the graphs would hold. From what I see here, that is true. – Mark Gerolimatos – 2015-03-03T20:56:02.213

How accurate is CURL's 'current rate' in the progress meter?

Answers