0

We have a deployment of security service on Kubernetes that needs to have the same private key to encrypt and decrypt API tokens that we send to clients.

At the moment, all the pods need to have the same private key. For example:

  • Pod A issues an API key to Alice
  • Alice then wants to access a secure resource
  • Alice makes the request that gets routed to pod B
  • If pod B doesn't have the same private key as pod A, then pod B will be unable to decrypt the key to verify, meaning Alice erroneously gets locked out

We'd like to find a way in Kubernetes to condition pod liveness on that pod having the same private key as the other pods. I realize this could create a death spiral situation if this fails, but that's fine. We do a blue-green deployment, so the live traffic would never get switched to the degenerate deployment until it was up and running.

My current thought is to write a script that runs as a liveness probe that compares the checksum of the private key file against the checksums of all the other pods, pulling the list of sibling pods from the Kubernetes internal APIs.

This is motivated by a case where the image pull policy was set to IfNotPresent and then an image with the same tag was updated between pod deployments, so we wound up breaking the service because different private keys were in different places. Obviously we could add checking this to our deployment checklist, but if there's a way to automate the enforcement of the policy through the actual deploy tool, I feel like that would be stronger.

I'm also willing to believe this approach of having multiple pods that all need to have the same key is the wrong approach for some reason. If there's a better way to achieve this goal, that would also be a useful answer to this question.

josephkibe
  • 101
  • 1

1 Answers1

1

Secrets are made for that:

Kubernetes secret objects let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image. See Secrets design document for more information.

How to create secrets:

Creating a Secret Using kubectl create secret

Say that some pods need to access a database. The username and password that the pods should use is in the files ./username.txt and ./password.txt on your local machine.

# Create files needed for rest of example.
echo -n 'admin' > ./username.txt
echo -n '1f2d1e2e67df' > ./password.txt

The kubectl create secret command packages these files into a Secret and creates the object on the Apiserver.

kubectl create secret generic db-user-pass --from-file=./username.txt --from file=./password.txt

secret "db-user-pass" created

How to use secrets:

Using Secrets

Secrets can be mounted as data volumes or be exposed as environment variables to be used by a container in a pod. They can also be used by other parts of the system, without being directly exposed to the pod. For example, they can hold credentials that other parts of the system should use to interact with external systems on your behalf.

Using Secrets as Files from a Pod

To consume a Secret in a volume in a Pod:

  1. Create a secret or use an existing one. Multiple pods can reference the same secret.
  2. Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].secret.secretName field equal to the name of the secret object.
  3. Add a .spec.containers[].volumeMounts[] to each container that needs the secret. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the secrets to appear.
  4. Modify your image and/or command line so that the program looks for files in that directory. Each key in the secret data map becomes the filename under mountPath.

This is an example of a pod that mounts a secret in a volume:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

Note that secrets reside in a namespace. They can only be referenced by pods in that same namespace. Full list restrictions is located here.

Piotr Malec
  • 271
  • 1
  • 5
  • I'm familiar with secrets, and gave that a thought. I'm worried that maybe someone updates the secret, then a pod gets restarted, and it pulls the new secret and is therefore out of sync with the other pods. Obviously you could version those to try to avoid that problem. Possible I'm chasing a chimera trying to do something that provides a more airtight guarantee. Thanks, though! – josephkibe Oct 24 '19 at 19:28