We've faced weird issue with one of volumeMounts of one of our deployments. Here are relevant manifests' snippets:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: xxx-media-pv-resource-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: cephfs
...
apiVersion: extensions/v1beta1
kind: Deployment
...
volumeMounts:
- mountPath: /opt/media/logs
name: xxx-media-log-storage
- mountPath: /shared-media-files
name: xxx-media-resource-storage
...
volumes:
- name: xxx-media-log-storage
persistentVolumeClaim:
claimName: xxx-media-pv-log-claim
- name: xxx-media-resource-storage
persistentVolumeClaim:
claimName: xxx-media-pv-resource-claim
...
We get read/write error: "Operation not permitted" when trying to, e.g. cat/echo from/to files in /shared-media-files although: a) can successfully touch/mkdir/chmod/rm there; b) can do everything in /opt/media/logs which is backed up by identical PVC (except name). Cephfs storageclass and corresponding Ceph cluster is a storage for whole k8s cluster and all k8s entities that use it except xxx-media-resource-storage behave as expected.
k8s master: v1.14.1
k8s workers: v1.16.0
cephfs-provisioner: v2.1.0-k8s1.11
Ceph: 14.2.1
We checked everything obvious like Ceph cluster health, free storage space, file permissions and ACLs, mount mode, PV/PVC descriptions, logs and didn't find anything special. As xxx-media-pv-resource-claim PVC accessMode is RWX, we tried to use it with another test pod and got same errors. We tried to google the issue as well but with no luck.
Please help to troubleshoot the issue.
