How can I pass secrets to a compromised container without the attacker being able to see them?

Question

The most common method of passing secrets to a docker container is through ENVs.

The problem is:

Imagine that your docker container is hosting a HTTP server that can have a security exploit (like any software), that will give almost command line access to an attacker.

Of course the attacker won't (maybe) be able to harm the host machine, but if he lists all the envs in the container he might see API keys used by the server, database passwords, etc.

So in the end, passing secrets as envs is not so safe after all...

Then you have the other secrets method that mounts secrets on the file system... The problem is that this same attacker can read these files too obviously.

So, in the end of the day, how to you store secrets in a container, taking into account that its running applications can get compromised?

Answer 1

Secrets passed to a container ARE for the use of the container process. The purpose of a ENV Secret is to expose the Secret to the process context run under that environment. All Secrets, regardless of how they are stored are for the process context of the container.

If a container is known to be compromised, your ENV secrets(and others in/passed to that container) are and should be handled as compromised.

Secrets should be handled with "The principle of least privilege (PoLP)" which refers to an information security concept in which a user is given the minimum levels of access – or permissions – needed to perform his/her job functions.

If you want to pass a NEW one-time-use secret for forensic usage to an environment, then an ephemeral container(k8s, alpha api) connecting to a running pod, with NEW secrets MAY allow you to perform some forensic collection and authentication during a triage operation.

• See: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

Answer 2

What you're asking for is impossible, unless you can use the secret only before opening yourself to attack and never need it again. If that works, you're OK; just make sure the secret is only present in the container during the non-vulnerable period (e.g. after the container starts but before it opens a listening port or makes an outbound request) and that the secret is then gone (removed both from however it was passed into the container, and from the process memory, and note that the latter is tricky since the compiler will optimize out a "useless" write) and unrecoverable before exposing any attack surface.

If that doesn't work, you need to re-think your requirements and maybe your whole design. You can't give something to a container without giving it to the attacker who owns the container and can see everything it contains. You can obfuscate the secret, but in the end, if the program running in the container could see / use it, the attacker who takes over that program can too. At best you make them do a little reverse engineering.

That doesn't mean there are no options, though.

First of all, try not to have vulnerabilities in the container. The security offered by containers should be defense-in-depth, not the full protection of your system. When evaluating a security boundary it is good to assume all earlier boundaries have failed, but that doesn't change the fact that, in most systems, the outer boundaries (the program running in the container, here) should be doing basically all the security work, and should be up to the task.

Part of that is operating under least privilege within the container itself. For example, if the process starts with moderately high (in the container's context) privileges and reads a secret that requires those privileges, the process can then irreversibly drop its privileges, such that an attacker taking over the process later won't be able to read the secret the same way (though if it's still in the process' memory, the attacker could potentially read that, so you might want to purge it from there if you can). That's not foolproof - it just means that the attacker needs to find a local EoP vulnerability to read it again - but it makes a single vulnerability probably insufficient.

You can deputize the holding of the secret. If your main concern is merely the attacker seeing the secret, rather than being able to use it, you can design a system that holds the secret outside of the container (or, slightly less securely, in a separate process of a separate user inside the container) and exposes some secured communication channel (e.g. a local (Unix) socket) that can be used by the container process to say "do <thing> with secret" and through which the secret will never actually be exposed, directly or indirectly. Of course, you probably want to limit the scope of <thing>, or else the attacker doesn't really have any need to know the secret at all, after all (it doesn't matter whether the attacker actually knows the password if she can just say "tell 'em the password" to your deputized secret-holder).

You might be able to make the secret short-lived. If it's for something that is only relevant for a short period of time - ideally shorter than the container lifetime, though this might help even if that's not true - you should rotate the secret as often as possible, and ensure each time a container starts that its secret will be invalidated as soon as possible. That way, an attacker who steals it will have limited use of it (or possibly none at all, if it has already expired).

Answer 3

So at a basic level, secrets used by a container will be vulnerable to attack by someone who compromises the container application, that's inevitable. The application has to be able to its secrets and the attacker will have the same access as the application. It's worth noting that this is no different to any other environment, an attacker who compromises an application on a bare metal server will get access to application secrets too.

Now to try and solve your problem, instead of asking "how do I stop the attacker getting access to my secrets" a different set of question is "How do I reduce the exposure" and "how do I reduce the usefulness of those secrets to the attacker" and here, there's some options.

For reducing exposure, you want to have container security software that can detect and alerts on the the attacker. There are commercial and open source options available for this.

The other piece is possible too. Firstly we can reduce the validity period of the secrets. Instead of having a key which lasts for days or months, we can mint a new secret which has a short lifespan, this reduces the window of attack.

Then we can try to tie secrets to specific sources. For example if I say "this secret can only be used when it comes from this pod in this k8s cluster", that secret becomes useless if exfiltrated to another system.