For TLS 1.3, you are limited to a few pre-selected named groups (RFC 7919) and you include the identifier of the group with your key share. See sections of the RFC.
In previous versions of the protocol like TLS 1.2, the server sent the p
and g
together with the key share in ServerKeyExchange
, again see the RFC. The server had to pick some values, either from Oakley group RFCs (RFC 3526) or a random value the server generated in advance using something like openssl dhparam 2048
.
To the second part of your question, absolutely not and they have nothing to do with each other. The key exchange part performed by DHE or ECDHE and the authentication part performed by RSA or ECDSA (or theoretically EdDSA) do not share keys. Note how they often use different cryptographic primitives in the same handshake (most popular on the internet now is ECDHE with RSA certificates)!
The fact that the a
and b
in DH are called "private keys" and g^a mod p
is called a "public key" is true, but a little confusing in the context of TLS. These are ephemeral, meaning single use, short term keys. They are the opposite of long term "identity" keys that you would use for authentication.
When you as a client connect to a server, the server has just created a new DHE key when you connected to it. Nobody has ever seen this key before, and when the handshake is over, it will be deleted. How would you use it for authentication? Who will you ask to confirm it?
Static-DH certificates used to exist (at least in theory) a long time ago. They don't exist anymore and are not allowed. They used long term DH keys for authentication and included the public key share in the certificate, but the certificate was signed by the CA using RSA. The key exchange performed ephemeral-static DH to confirm the server. This did not provide PFS and died out in favor of RSA key exchange (which also did not provide PFS) and then ECDHE+aRSA, which finally provided PFS.
Outside the context of TLS, protocols like those based on the Noise Protocol Framework, like WireGuard, use long term ECDH keys for authentication, together with different ephemeral ECDH keys used for key exchange, thus reusing code (less code to review). TLS doesn't do that because you need to confirm the validity of the certificate chain, and for that you need offline signature validation, which you can't do with Noise (as far as I understand, please someone correct me if I'm wrong).