Better way to use Perf on a Program in Kubernetes/Docker?

Question

I am running Kubernetes on CentOS 7, and it doesn't seem that the version of perf is namespace aware.

If run perf on the PID on the host node (host node found with kubectl describe pods --namespace) I get an error about symbols not being found. This seems to be because it looks for the path of the file relative to the container but on the host node's filesystem.

If I copy the exe (a Go exe that includes the symbols) to the expected path on the host node (either with kubectl cp ..., or by finding the file under overlay2 in /var/lib/docker) then perf top -p <pid> works from the host host node since it can find the same symbols that match the exe in the container.

Is there a better/cleaner way to run perf against a process running in a container with a CentOS 7 host?

References:

http://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/ section "The hack"
Kernel Recipes 2017: Using Linux perf at Netflix (SlideShare) - Slide 67.

Why not install `perf` inside the container and run it there? — Matt Zimmerman, Sep 05 '19 at 20:59

score 1 · Answer 1 · answered Feb 11 '20 at 21:53

Since Kubernetes 1.17 it is possible to Share Process Namespace between Containers in a Pod. It seems like this is what you are looking for.

Process Namespace Sharing is enabled using the shareProcessNamespace field of v1.PodSpec. E.g.:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  shareProcessNamespace: true
  containers:
  - name: nginx
    image: nginx
  - name: shell
    image: busybox
    securityContext:
      capabilities:
        add:
        - SYS_PTRACE
    stdin: true
    tty: true

The Kubernetes Docs I linked give a little more detail on how to work with this sidecar approach.

Better way to use Perf on a Program in Kubernetes/Docker?

1 Answers1