I'm running kubernetes on Ubuntu 18.04.1, running some of my pods with mounts via ISCSI. I've noticed recently (maybe due to improperly unmounting the iscsi volume or for other reasons) that at least one of the iscsi extents (ext4 fs) is sometimes corrupt. In an effort to try and fix this, I'd like to change the mount to journaled instead of ordered. I see this in the node's syslog:
Dec 29 11:45:29 mira-b systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.xx.yy:3260-iqn.2005-10.home.mediastore:container-lun-0.
Dec 29 11:45:29 mira-b kubelet[27086]: I1229 11:45:29.988889 27086 operation_generator.go:506] MountVolume.WaitForAttach succeeded for volume "container-data-folder" (UniqueName: "kubernetes.io/iscsi/192.168.xx.yy:iqn.2005-10.home.mediastore:container:0") pod "container-65d6d65b75-z5l94" (UID: "41ef97fd-0ba2-11e9-ba38-0cc47abd5a26") DevicePath "/dev/disk/by-path/ip-192.168.xx.yy:3260-iscsi-iqn.2005-10.home.mediastore:container-lun-0"
Dec 29 11:45:29 mira-b kubelet[27086]: I1229 11:45:29.988951 27086 operation_generator.go:527] MountVolume.MountDevice succeeded for volume "container-data-folder" (UniqueName: "kubernetes.io/iscsi/192.168.xx.yy:iqn.2005-10.home.mediastore:container:0") pod "container-65d6d65b75-z5l94" (UID: "41ef97fd-0ba2-11e9-ba38-0cc47abd5a26") device mount path "/var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.xx.yy:3260-iqn.2005-10.home.mediastore:container-lun-0"
Dec 29 11:45:29 mira-b kernel: [ 389.683464] EXT4-fs (sdi): mounted filesystem with ordered data mode. Opts: (null)
That last line tells me that ext4 fs is being mounted as ordered, but in the k8s docs I don't see any way to specify how to mount the FS, only "fs: ext4" in the iscsi volume options.
inside the pod/container, i see this
/dev/sdi on /config type ext4 (rw,relatime,stripe=256,data=ordered)
Is there a way to explicitly have kubernetes mount the ext4 fs using data=journaled instead of ordered? FWIW, my ISCSI backing store is ZFS, I've already enabled sync writes for the dataset that extent lives on, but to rule anything else out I'd like to have the container also mount journaled to try and ensure any writes are completed before any power-off/unmmounts, etc.
PS: Kubernetes version 1.13.0
Update: Forgot my yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: pod
spec:
replicas: 1
template:
metadata:
labels:
app: pod
spec:
hostNetwork: true
volumes:
- name: pod-data-folder
iscsi:
targetPortal: 192.168.xx.yy
iqn: iqn.2005-10.home.mediastore:pod
lun: 0
fsType: ext4
looking at @VAS's answer, it seems like mount options aren't valid when specifying iscsi volume inline, and that it needs to be provided as a PV similar to NFS mounts. I'll give that a try.