What are the security implications of capabilities in Kubernetes pods?

Question

We have a Kubernetes deployment with an application that need to be on a VPN. We implement this requirement by running openvpn-client in a sidecar container within the pod with elevated capabilities:

securityContext:
  capabilities:
    add:
      - NET_ADMIN

We'd like to better understand the impact of this, and how exposed we'd be if this container were compromised. We want to be confident that code exec in this container couldn't view or modify packets or network configuration in other pods, or on the host node.

My current hypothesis is that since each pod has an isolated network namespace, giving CAP_NET_ADMIN to a container in the pod just provides the capability within that namespace.

However, I haven't been able to find any documentation that's definitively discusses the impact of using securityContext to assign capabilities to containers. There's a few pieces of documentation - outlined below - that strongly imply that Kubernete / Docker will provide sufficient isolation here, but I'm not 100% certain.

The pods documentation on resource sharing [1] gives a hint here:

The applications in a Pod all use the same network namespace (same IP and port space), and can thus “find” each other and communicate using localhost. Because of this, applications in a Pod must coordinate their usage of ports. Each Pod has an IP address in a flat shared networking space that has full communication with other physical computers and Pods across the network.

The networking documentation on the network model has this to say:

Kubernetes IP addresses exist at the Pod scope - containers within a Pod share their network namespaces - including their IP address. This means that containers within a Pod can all reach each other’s ports on localhost. This also means that containers within a Pod must coordinate port usage, but this is no different from processes in a VM. This is called the “IP-per-pod” model.

Finally, I note that pod.spec.hostNetwork is configurable, and defaults to false:

$ kubectl explain pod.spec.hostNetwork
KIND:     Pod
VERSION:  v1

FIELD:    hostNetwork <boolean>

DESCRIPTION:
     Host networking requested for this pod. Use the host's network namespace.
     If this option is set, the ports that will be used must be specified.
     Default to false.

[1] https://kubernetes.io/docs/concepts/workloads/pods/pod/#resource-sharing-and-communication

[2] https://kubernetes.io/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model

score 1 · Answer 1 · answered Dec 23 '19 at 13:38

For this I think it's quite probably that a rogue process running with that capability would be able to affect the underlying host, if only indirectly.

From this man page we can see that CAP_NET_ADMIN provides quite a few privileges, including interface configuration

       CAP_NET_ADMIN
              Perform various network-related operations:
              * interface configuration;
              * administration of IP firewall, masquerading, and accounting;
              * modify routing tables;
              * bind to any address for transparent proxying;
              * set type-of-service (TOS)
              * clear driver statistics;
              * set promiscuous mode;
              * enabling multicasting;
              * use setsockopt(2) to set the following socket options:
                SO_DEBUG, SO_MARK, SO_PRIORITY (for a priority outside the
                range 0 to 6), SO_RCVBUFFORCE, and SO_SNDBUFFORCE.

These permissions are restricted to the namespace of the process, but allowing an interface to go into promiscuous mode could allow for traffic to be sniffed from other interfaces on the host, depending likely on the CNI configuration. Also as noted in this white paper there have historically being some breakouts and privescs related to NET_ADMIN.

What are the security implications of capabilities in Kubernetes pods?

1 Answers1