Clarifying self-signed certificates vs root certificate authority

Question

I hope someone can help me understand some fundamentals on SSL certificates that I've had trouble picking up from docs, Wikipedia, and just about everywhere else on the internet.

I'm working on an application that communicates with another on a separate network. Each application acts as both a client and a server in bi-directional communication. They will use REST services over HTTPS with each network's firewall opened explicitly for the other. All SSL certificates will be self-signed. My application is written in Java but I'm not sure about the other.

I believe I have two options to establish the HTTPS connections with self-signed certificates:

1. Each application's underlying Framework (e.g. Java) installs the other's self-signed certificate, resulting in that framework, and not necessarily the wider OS, trusting the certificate
2. Each server installs the other's certificate as a root Certificate Authority and therefore trusts any self-signed certificate produced by the other server. A framework like Java should acknowledge this OS-level root Certificate Authority but a web browser on that server might not as it maintains its own database

The server I'm working on has the following files already generated: server.crt and server.key. I believe these are generated using openssl genrsa. I understand that the .crt is the public certificate and the .key is its private key counterpart. My main question is

1. Can the .crt be treated as either a framework-level certificate or an OS-level root Certificate Authority by the other server (i.e. do I get to choose how I install it)? In my research online this has been very confusing as many of the terms seem overloaded. I can't tell if I'm working with a Certificate Authority or just a regular certificate.
2. If the answer to 1 is no, what kind of file do I need to generate from the existing files to install just the self-signed certificate?

Any help much appreciated, and please assume that providing links to Wikipedia pages will not help me much.

Answer 1

You should see the certificate infrastructure (that is to say the PKI, Public Key Infrastructur) as a web of trust.

When you are communicating with multiple persons, which wants to mutually authenticated themselves (and their websites), you first choose a common trusted person. This third person becomes a "Trusted Third Party" and will be called the "Certificate Authority" (CA) for the purpose of certificate management.

The CA will create a master key, also known as root CA key. The private key will be kept secret by the TTP at all cost. The public key will be embedded into a certificate, and this certificate will be signed by the public key of the CA. This means the certificate is self-signed and you can verify the signature with the key inside it.

Let's assume now that you want to access Alice's website, but you want it to really be Alice's website and not another person that claims to be her. Alice is part of you community and therefore contacts the CA you agreed to use. She sends the CA a CSR (Certificate Signing Request) which basically is a public key. The CA do the process of verifying that Alice is Alice (for example, the responsible person meets Alice in person and get the CSR from her hands). When the CA is sure Alice is Alice and the CSR belongs to her, the CA can create a certificate in the name of Alice and sign this certificate with the CA private key.

When you connects on Alice's website, you ask for the certificate. Once you get the certificate, you want to verify it's the good one. You can see in the certificate that it has been issue by a CA. If you have the CA key you can verify the signature. If you trust the CA is doing its job correctly, then you can trust that this certificate is a good one and since the CA trusts it authenticated Alice in the first place, then you can trust that this certificate belongs to Alice.

You can then be sure that encrypting a message with this public key will only be read by Alice who has the corresponding private key. You are also sure that anything singed by this certificate is a signature made by Alice.

If Alice wants to authenticate you in return, then you need to do the same things (get a certificate signed from the CA, since Alice trusts the CA as well she will trust this certificate)

In your case, since it seems you master all the endpoints, you could simply create two self-signed certificates (which are in fact two root CA certificates) and exchange the public parts to the other endpoint. Each endpoint will use their own private key to sign outgoing requests and decrypt incoming messages, and the other endpoint certificate to encrypt outgoing message and verify incoming messages.

If you happen to have a lots of endpoint, it could be worth to create your own CA. You generate a root CA certificate then you sign certificate for each endpoints. At each endpoint you can install the public root CA cert. Then each time a endpoint connects to another, the contacted endpoint can send its certificate and the caller can verify it against the root-CA cert.

The first method has the advantage of being straight forward, but compromising of one certificate implies to exchange it for a new one on all endpoints. The second method has the advantage to be scalable, one endpoint does not have to know how many other endpoints exists since it can validate the certificate on the fly anyway. Moreover there are way to manage key compromising with Certificate Revocation Lists for example.

Answer 2

The two options you mention are almost correct: However, you can (and should) install self-signed certificates without them being Certificate Authority certificates.

The difference between a self signed cert and a CA cert is that a CA certificate is a special self-signed certificate with its "basicConstraints" set to "CA:true" (usually with the critical flag set). Additionally, the CA cert's keyUsage sequence lists "cRLSign" and "keyCertSign" as purposes which should not present in a regular (non-CA) certificate.

Now to your two questions:

1. You can use it as a framework-level or OS-level certificate to your liking, but do not confuse self-signed certificates with CA certificates (see above). Since the certificate seems do be a special purpose certificate for your application only, I would recommend to use it as framework-level cert. Also, installing a trusted OS-level cert might require administrator privileges which not all communication partners might have.

2. N/A (since the answer to 1. is yes :) )

On a side note: The reason why the distinction between CA and non-CA self signed certificates is relevant is that you should not allow a self-signed certificate, whose purpose is to validate a communication partner, to act as a certificate authority since this might open up additional security implications (i.e. MITM attacks using said certificate as trusted authority).

Answer 3

Here your are trying to authenticate and trust both the networks while acting as bi- directional. The trusted authority certificate is different/same and CA issuing certificate may be different or the same it doesn't have any problem over on it.

Note: Don't use self-signed certificates for SSL communications and follow the standard RFC rules.

If you want to establish communication between two networks there are multiple ways. 1) Tunnel Establishment 2) PTP 3) SSL based: a. Just create two different CA's and place it across networks and each CA certificate will be placed with each network. Now each CA should issue the certificate and it will be validated by the trust and the communication will be more secure now.

Answer 4

What the application should do after setting up the connection is to check the validity of the received cert. You could do this with a nod to pki, ie.

1. you have CA certs on the application that can verify the issuer of the received cert

or

2. A received cert's cert issuer i.e. itself.

A lot of this depends on if you're rolling your own library or depending on an api.