How to benchmark ChaCha20-Poly1305 capable websites using Apache Benchmark (ab) tool?

Question

I'm trying to use ab to do some performance benchmarks of my website after having made some performance tweaks.

Specifically, I'd like to test the difference in performance between the following cipher suites - all supported by my website:

ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-ECDSA-CHACHA20-POLY1305 

The three commands that I've tried are:

ab -l -n 1000 -c 10 -H "Accept-Encoding: gzip, deflate, br" -Z ECDHE-RSA-AES128-GCM-SHA256 https://bytes.fyi/
ab -l -n 1000 -c 10 -H "Accept-Encoding: gzip, deflate, br" -Z ECDHE-ECDSA-AES128-GCM-SHA256 https://bytes.fyi/
ab -l -n 1000 -c 10 -H "Accept-Encoding: gzip, deflate, br" -Z ECDHE-ECDSA-CHACHA20-POLY1305 https://bytes.fyi/

The first two work fine, but the third generates the following error:

error setting cipher list [ECDHE-ECDSA-CHACHA20-POLY1305]
1995798240:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl_lib.c:1385:

I think my versions of ab and openssl are both up-to-date enough to support the test:

pi@pi3:~ $ which ab && ab -V
/usr/bin/ab
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

pi@pi3:~ $ which openssl && openssl version
/usr/bin/openssl
OpenSSL 1.1.0f  25 May 2017

The docs for Apache Benchmark don't give much detail on how to check/modify the available cipher suites that can be specified:

-Z ciphersuite
Specify SSL/TLS cipher suite (See openssl ciphers)

I think the above implies that I should be able to use any of the cipher suites listed by the openssl ciphers command.

All three of my target cipher suites are indeed listed, so I'm confused why my ab test is failing for the ECDHE-ECDSA-CHACHA20-POLY1305 suite.

Here's some output to show my target suite is supported by my version of openssl:

pi@pi3:~ $ openssl ciphers -v | grep ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD

Answer 1

Answering my own question now that I've solved this, in case it's useful to others...

It turns out that my version of ab was too old to (properly) support OpenSSL 1.1.0, as Konstantin Kolinko advised me (through the httpd support mailing list):

Looking at http://svn.apache.org/r1757674 (Thu Aug 25 12:53:03 2016 UTC) and history of httpd/httpd/branches/2.4.x/support/ab.c file that was changed in that revision, I think your version of AB does not support OpenSSL 1.1.0 at all, as support for 1.1.0 was added by later revisions of that file,

http://svn.apache.org/viewvc?view=revision&revision=1787728 "Support OpenSSL 1.1.0"

In the end I had to build the latest trunk code of the whole Apache httpd, to get a modern, working version of ab. I just built it in my home folder, as I wasn't actually going to use the other elements of httpd at all:

# apply any updates and install pre-requisites
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install libapr1 libapr1-dev libaprutil1 libaprutil1-dev libpcre3 libpcre3-dev subversion autoconf libtool libtool-bin

# move to home folder and create a folder where to build into
cd ~
mkdir httpd-install

# get the latest source and unzip it (to ~/httpd-trunk)
wget https://github.com/apache/httpd/archive/trunk.zip
unzip trunk.zip
cd httpd-trunk

# get the latest APR source and put it in '[httpd source root]/srclib/apr' (required for the build)
svn co http://svn.apache.org/repos/asf/apr/apr/trunk srclib/apr

# configure the things
chmod +x buildconf
./buildconf
./configure --prefix=/home/pi/httpd-install/ --with-included-apr

# make and install
make
make install

# create a symlink to our bleeding-edge version of ab, so we can just call it using 'ab'
sudo ln -s /home/pi/httpd-install/bin/ab /usr/bin/ab

Not exactly simple, then... but it did work. I now have a bleeding-edge version of ab, which seems to work fine even with modern cipher suites like ECDHE-ECDSA-CHACHA20-POLY1305:

pi@pi3:~ $ ab -l -n 1000 -c 10 -k -H "Accept-Encoding: gzip, deflate, br" -Z ECDHE-ECDSA-CHACHA20-POLY1305 https://bytes.fyi/
This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bytes.fyi (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        bytes.fyi
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-ECDSA-CHACHA20-POLY1305,256,256
Server Temp Key:        
TLS Server Name:        bytes.fyi

Document Path:          /
Document Length:        Variable

Concurrency Level:      10
Time taken for tests:   6.876 seconds
Complete requests:      1000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      10464567 bytes
HTML transferred:       8983567 bytes
Requests per second:    145.42 [#/sec] (mean)
Time per request:       68.764 [ms] (mean)
Time per request:       6.876 [ms] (mean, across all concurrent requests)
Transfer rate:          1486.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   31   9.6     32     110
Processing:    16   37   9.4     36     116
Waiting:       14   33   8.5     34      93
Total:         30   68  14.2     69     153

Percentage of the requests served within a certain time (ms)
  50%     69
  66%     72
  75%     73
  80%     74
  90%     77
  95%     87
  98%    103
  99%    116
 100%    153 (longest request)