3

We would like to harden our Docker Image and remove redundant software from it. Our Devs and Ops asked to keep some Linux tools used for debugging on the containers running on our Kubernetes Prod environment.

I’ve read this post: https://www.digitalocean.com/community/tutorials/how-to-inspect-kubernetes-networking

And it made me wonder, is it possible to run commands that exist only on the host, on a container (which those commands have been removed from)?

If so is there a difference between commands that have been removed from the container than ones that the user don’t have permissions to run?

P.S. How do the tools in the above mentioned post work?

Thanks for the help! :)

whiteberryapps
  • 135
  • 3

1 Answers1

5

Yes this is possible using nsenter or similar tools, depending on what you want to do you can also use Docker tooling.

A Docker container is just a namespaced Linux process, so using namespace aware tools you can execute a command from the host inside a given namespace.

For example, say you want to use tcpdump to diagnose a problem with networked nginx web site.

you can run docker run -d --name nginxtest nginx, to start the container, then run ps -ef | grep -i nginx and get the PID of the nginx process.

Once you've got that you can use something like sudo nsenter --target [PID] --net /usr/sbin/tcpdump to run TCP Dump from the host inside the network namespace of the nginx container.

Another option is to use a sidecar container which shares namespaces with a main container. So for example you can run docker run -it --net container:nginxtest raesene/alpine-containertools /bin/ash to run a new container which shares the network namespaces of the nginxtest container.

For mount namespace, it's a little more complex but still possible. you can view/edit files via the proc filesystem. So for example vi /proc/[PID]/root/etc/passwd would allow you to view/edit the /etc/passwd file from the running container . With this you do need to be careful of file permissions

Rory McCune
  • 60,923
  • 14
  • 136
  • 217
  • Great thanks for the answer! What about doing the same for a container running on Kubernetes? Should we run ‘nsenter’ from the worker node or is there another option doing that with kubctl? – whiteberryapps Aug 14 '20 at 08:17
  • so on Kubernetes that's a little more interesting. there is a feature called "ephemeral containers" which is currently in Alpha which addresses this kind of requirement https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ . You could also go to the worker node but obviously that's not a great option , especially in larger clusters. – Rory McCune Aug 14 '20 at 08:23
  • So if I understand correctly the reason to use an Ephemeral Container is to reduce the attack surface, as the debugging container will be destroyed after usage. Having said that, as Ephemeral Containers is an Alpha feature and is not production ready. Would adding a sidecar container with limited amount of read commands used for debugging, which share the same process namespace as the application (while the application’s container won’t have any command available) could be considered a current best practice verses leaving those commands to exist on the application container? – whiteberryapps Aug 14 '20 at 14:37
  • So we're probably getting a little off piste for Sec.SE here, A sidecar can help I think, but I'm not a day-to-day k8s admin. I'd say you might want to get some more "on the ground" answer from the Kubernetes slack, that's usually a good place to ask questions :) – Rory McCune Aug 15 '20 at 15:38