Mix and match RSA and ECDSA in TLS

Question

Suppose TLS client and server want to communicate, authenticating both parties. They each have sufficiently trusted certificates, but one corresponds to an ECDSA private key and the other corresponds to a RSA private key, or maybe ECDSA private key with a different curve (not sure if this makes any difference). I am not specifying the algorithms used to sign those certificates, but please say if this is relevant. Would these parties be able to communicate - either at all, or, specifically, with the client being a mainstream modern browser?

Answer 1

YES -- there is no necessary connection between the types of keys and certs used by TLS server and client for authentication (nor between the Certificate Authorities and trust decisions).

For TLS versions through 1.2 (and SSLv3, which no one should be using now), the server key-and-cert must match the key exchange portion of the ciphersuite used, which the server selects from among those offered by the client. See the table beginning about a page into rfc5246 section 7.4.2 and similar in earlier versions. In practice this is implemented the other direction: when choosing the ciphersuite the server will only consider those offered by the client and for which it has a suitable key-and-cert. (In cases where the server is mistakenly configured with no key-and-cert(s), which is fairly easy in Java where usually you configure only a keystore file and people who don't follow correct instructions can easily create a keystore file that doesn't actually contain a key-and-cert, this causes the server to reject all connections with an exception or error saying 'no cipher overlap', which some people find confusing and ask about on Stack.)

For ECDSA cert, the client may (and most do) specify the curves it implements using extension 10 defined in rfc4492 section 5.1.1 and modified by 7027 for Brainpool as linked by the ietf website at the top, in which case the server is only permitted to use a cert (and if applicable chain) using the specified curve(s), and thus in practice will select a ciphersuite requiring an ECDSA cert only if its ECDSA cert (chain) uses such (a) curve(s). The client similarly can constrain point formats used in the server cert(s) using extension 11, but in practice this is usually not an issue.

In TLS1.2 only, for both RSA and ECDSA (and DSA as well) the client can specify signature_algorithms extension 13 defined in rfc5246 section 7.4.1.4.1 which constrains both the public-key algorithm(s) RSA,ECDSA,DSA and the hash(es) used by the server cert chain; see the text at the end of 7.4.2 in rfc5246 only (not earlier versions). Before 1.2 there was no explicit constraint on CA/chain certs above the server cert (except for the rarely used static DH_{RSA,DSS} and ECDH_{RSA,ECDSA} keyexchanges), but it was a fairly common convention to be homogenous, i.e. a server RSA cert would be issued by an RSA CA and a server ECDSA cert would be issued by an ECDSA CA (either the same curve or a stronger one, in practice almost always the two curves then in NSA's Suite B, P-256 and P-384).

For TLS 1.3, keyexchange and server cert is no longer tied to ciphersuite. Instead client always specifies signature_algorithms extension 13 (but using mostly different values than 1.0-1.2) and may also specify signature_algorithms_cert extension 50 as described in rfc8446 section 4.2.3. The supported curves are now limited to the three FIPS-186 prime curves (P-256, P-384, P-521) and the two Bernstein curves (ed25519, ed448) -- which were pretty much the only ones anybody used anyway -- and there is no option for point format; the originally-X9 curves must use uncompressed and the Bernstein curves use the only format defined for them.

Lastly, for some application protocols using TLS notably including HTTPS and thus web browsers, the server cert must match the hostname expected by the client. This is usually a fully-qualified domain name (FQDN), and since the TLS connection is made at the IP-address level, the ClientHello may contain a server_name extension (defined in rfc3546 for 1.0, rfc4366 for 1.1, rfc6066 for 1.2 and 1.3) to guide the server in choosing the key-and-cert to use.

The client cert on the other hand is controlled mostly by the CertificateRequest message sent by the server. In all protocols if the server doesn't send CertificateRequest the client does not authenticate at all. In 1.0 and 1.1 (and SSLv3) CertificateRequest specifies only the supported (acceptable) public-key algorithms, which need not be the same as the one(s) for the server's own key-and-cert (chain), and possibly a list of Certificate Authorities (CAs) that the server prefers the client use, which again need not be the same as used by the server's cert chain. In 1.2, the server also specifies a signature_algorithms field which constrains the client cert chain in the same way the extension in ClientHello constrains the server cert chain. For ECDSA in 1.0 through 1.2, there is no constraint on curve, apparently on the theory that servers are never constrained e.g. embedded/IoT(!), but the server may constrain point formats using extension 11 in ServerHello (before CertificateRequest) which AFAICS provides no benefit. In 1.3, the server must specify signature_algorithms and may specify signature_algorithms_cert, as well as certificate_authorities (now an optional extension instead of a possibly empty field); see rfc8446 section 4.3.2 including the text about 'prior versions'.