1

When running following command for cluster down in Kubernetes, I am getting following error:

KUBERNETES_PROVIDER=ubuntu ./kube-down.sh

rm: cannot remove ‘/var/lib/kubelet/pods/16981b98-a3bb-11e5-99fb-00505622b20d/volumes/kubernetes.io~secret/default-token-0i2n6’: Device or resource busy

I tried to remove it forcefully but then also its not getting removed.

Community
  • 1
maddy25
  • 11
  • 1

2 Answers2

1

I think this was fixed by https://github.com/kubernetes/kubernetes/pull/17315 You might want to check if you have the latest cluster/ubuntu/* files.

Eric Tune
  • 155
  • 5
1

In case you're wondering what the underlying mechanics of such an error are, seeing a Device or resource busy typically means there's a kernel-held "lock" on something at that point in the file system and the operation for unlink-ing (that is the syscall that the rm command uses to request this action from the kernel) and is prevented, with an error condition (EBUSY). You can learn a lot more about what syscalls are used by running a command through strace if you want to see all syscalls involved (including the return codes from syscall invocations).

That path looks to me where kubelet manages pod-specific "filesystem-esque" parameters, in this case, what is happening is that every namespace has a default service account that then gets a default token, and that token is transparently mounted inside the pod container's file system. That path mentioned in the error, is the host-level filesystem location of the pod's "internal" filesystem. So in this case, the "lock" in question is a mount-point.

Basically that script is trying to removing that path (perhaps clean-up after itself) and it's still in-use (perhaps the pod was still running), and the script or some command it's invoking is not doing proper error-checking/handling, so you end up seeing that error instead.

Aleks K
  • 46
  • 1