Can we restrict cipher suites using server certificate?

Question

Can we restrict cipher suites (such as RC4) using server certificate?

Answer 1

A TLS cipher consists of a part defining the authentication of the certificate (i.e. ECDSA, RSA...), a part defining the key exchange (ECDHE, DHE...) and a part about the kind of symmetric encryption and the associated HMAC, i.e. RC4+SHA1, AES128+SHA256 etc.

From these parts only the authentication part depends on the type of certificate and everything else is independent. This means that you are not able to restrict the use of specific symmetric encryption within the certificate, like denying RC4. This can only be done by configuring either client or server to not include such ciphers in their SSL configuration.

Answer 2

Update: this answer applies to TLS through 1.2 only. TLS 1.3 in 2018 changes the handshake radically, and now keyexchange, authentication, and symmetric (data) encryption/MAC are indepedent with only the last (plus KDF) determined by the 'ciphersuite'. And RC4 is no longer permitted at all, mooting this Q.

TLDR: in limited cases yes, but don't

First, key-exchange and authentication aren't really independent in SSL/TLS. Although there are sometimes a few choices available, many combinations you could reasonably choose do not exist as ciphersuites. (Unlike for example SSH, where the pieces are negotiated separately and you can get any combination as long as both sides support each piece.) In particular, (classic Zp) Diffie-Hellman Ephemeral (DHE) keyexchange can be authenticated with RSA or DSA (spelled DSS) but not ECDSA, and Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) keyexchange can be authenticated with RSA or ECDSA but not DSA. And some suites do NO authentication at all, called 'anonymous', although they are rarely used and frequently prohibited precisely because they are unauthenticated.

Second, there are many combinations of keyexchange including auth (if any) with symmetric ciphers and HMAC-and/or-hash -- but not all. For one, the ECC suites with ECDHE ECDH(static) or ECDH_anon don't use any of the 'export' (40-bit) ciphers or single-DES or SEED, presumably because by 2006 (rfc4492) there was no longer a need for these weak ciphers. Also and relevant here, there are no suites that use DSA/DSS authentication with RC2, RC4, or IDEA. My guess is DSA/DSS was largely to please the US government, which in the 1980s and 1990s had a thing about not paying royalties to RSADSI which back then enforced the RSA patent pretty vigorously. But USG also had a thing about using only ciphers 'approved' by NBS-later-NIST, and DES was approved but RC2 RC4 IDEA were not. (Neither was RC5, but RC5 wasn't used in SSL-later-TLS in the first place.)

SO: if the server uses a DSA key and certificate in standard SSL/TLS, it cannot negotiate RC4.

But there are downsides:

Certification: this is the big one. I've seen no public CA (Verisign, Comodo, Godaddy, etc.) that offers DSA certificates -- although I could have missed one. I know USG used to run their own internal PKI using DSA, but I don't know if they still do, and even if they do you probably aren't eligible for a cert from them and anyway their roots are not in the truststores of non-USG systems and devices. You can of course easily create a self-signed DSA cert, or easily enough create your own CA that issues a DSA cert, but establishing (and even maintaining) trust for a self-cert or self-CA is frequently a good deal of work and sometimes just infeasible.

DSA (parameters) size: (classic) DSA and DH are subject to nearly the same attacks as RSA, and although neither 1024-bit factoring or 1024-bit dlog has been reached in the open community, they are widely believed to be feasible with fairly modest improvements in technology. The Logjam researchers estimated (in late 2015) breaking DH-1024 to be '[p]lausible with state-level resources'. The same logic applies to DSA-1024, although it is less common to share DSA parameters than for DH(E), which might reduce the motivation of the attacker. At any rate, standards have mostly required minimum 2048 bits for adequate safety margin for RSA DH and DSA since 2014, driven substantially by NIST.

But the original DSA standard FIPS-186(-0) required the prime p defining the (outer) group to have size 512 to 1024 bits in steps of 64. In 2000, 186-2 change notice 1 eliminated the smaller sizes leaving only 1024. But it took until 2009 for NIST to finalize 186-3 adding p sizes 2048 and 3072 (with q 224 or 256 and 256 respectively). Given the way applications build on 'core' functionalities like crypto, and software is developed, tested, acquired, and deployed, only in the last year or two did DSA-2048 mostly work and it still is far from universal. As one example, the Sun-now-Oracle version of Java implemented 186-3 sizes only with version 8 in 2014. OpenSSL implemented large p as an extension long ago, but the 186-3 q sizes only with 1.0.0 in 2010 -- and 1.0.0 was a major version with API changes that took some applications years to adapt to and systems more years to fully deploy.

DH never had such a limitation officially (although see below); some specific parameters were standardized that are now too small, but they were only one option among several. RSA did not have any standardized limitation I know of; early implementations were often limited, sometimes to only 1024 bits or not much more, but this was always treated as an implementation limit to be raised as soon as computer power allowed it, and AFAIK all implementations have supported at least 4096 for many years now.

DHE status and size: due to the relationship between keyexchange and authentication noted above, a DSA key&cert can only be used with DHE key exchange or SRP, and in practice effectively nobody uses SRP for SSL/TLS. Much software until fairly recently implemented DHE with fixed or default and sometimes inadequate parameters; it was only after Snowden that a substantial number of people (both users, clients like browsers on behalf of users, and server admins) started wanting ephemeral keyexchange (DHE or ECDHE). Since this was only a few years ago, and most information and blogs and 'hints' and 'tricks' you find searching the InterWeb are usually years to decades out of date if not wholly fictional, it's very easy to set-up DHE insecurely. While in principle ECDHE could be screwed up similarly, in practice many implemented only the former-SuiteB curves P-256 and P-384, plus sometimes P-521. This means if you found advice saying ECDHE=good, and managed to get ECDHE to work at all, it was secure.

Suncle Java before 8 was even worse here: JCE restricted DH to the DSA (186-0) sizes apparently as a side-effect of reusing DSA parameter generation for DH, which is silly enough -- see numerous Qs on SO for client 'Could not generate DH keypair', but JSSE server chose nonexport DHE 768 bits (and hardcoded to boot) on the entirely spurious ground that 768 is twice the kx=RSA premaster size of 384. (First of all different KX can have different premaster, that's the explicit reason for the two-step derivation premaster->master->working. But even if you had a valid reason to want 384 strength, that would mean the subgroup is 768, NOT the group!)

So in sum: don't do this. Avoid RC4 by prohibiting RC4 directly, the way you should. Also avoid the export and single-DES suites by prohibiting them directly if necessary -- although a fully conforming implementation of at least 1.1 and 1.2 respectively already doesn't allow them.