0

Hi ServerFault Community.

I am in need of your help with a rather annoying edge case problem, which I hope can spark an universally helpful solution:

I need a solution, similar to a sidecar container, but one that:

  • doesn't need to be declared in advance
  • doesn't require a restart of the main app container
  • doesn't require root on the host
  • can be plugged in without persistent changes to the app container, or the changes can be reverted easily once I am done
  • ideally would let me monitor/mount the entire filesystem of the app container on another debugging container, and execute commands/tools that would be only in this debugging sidecar container

My particular issue: I have a long running docker container of a legacy system that has only the app running in it and no usable shell.

Some of files created by the app are not in a persistent volume, and I need to get access to them, on the spot, and in the right time, for debugging purposes with additional tools/commands that are not installed in the container, nor the host.

It is a PROD system, and the issue is elusive enough that only this container seems to be fit for debugging that particular case. I can't restart the container with updated config to place the required filesystem location in a persistent volume, for security reasons.

I already tried copying everything from the container to the host, and even used the root on the host to access the container filesystem structure visible on the host, but this solution does not give everything, and is not feasible in the long run either way.

It is not the first time I stumbled upon a situation where such solution could be helpful, so I would be grateful for solutions to any mainstream container platform, and I guess others have faced this problem too.

A similar problem has already been posted: how to export the filesystem from one container to another but it is not quite the same as my case.

Any solutions that would fit the above points are very welcome.

Please help.

stack3r
  • 101
  • 1
  • Have you considered just copying the files out of the container to another location? It is pretty easy to export the files from a running container. `docker export container_name | tar -C /dest_path -x` would basically copy everything from the container to some path for debugging purposes. – Zoredache Aug 04 '22 at 20:58
  • You could also possibly just switch into the 'merged' directory on the host, and inspect things there. You can see the directory with `docker container inspect container_id | jq '.[] | .GraphDriver.Data.MergedDir'`. – Zoredache Aug 04 '22 at 21:07
  • Hi @Zoredache. Thank you for your feedback. I can see why you mention both of these options, and I have already tried them. But I need to monitor short lived temp files, for example with inotify. I need a solution that does not rely on root on host, and allows "live" access to FS. Maybe I can find a copy-on-write solution, but I would prefer something like a transparent sidecar with extra tools installed in it instead. – stack3r Aug 09 '22 at 07:39
  • This sounds very similar to what [kubectl](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container) debug can provide. It allows adding ephemeral containers to a running Pod. It's beta as of Kubernetes 1.23, I suspect it's not available in the earlier version. – Abhijith Chitrapu Aug 10 '22 at 11:11

0 Answers0