OpenShift Origin and Ceph persistent volume

Question

I have installed OpenShift Origin from latest ansible install. (CentOS 7 - 3 masters and 7 nodes)

[root@master-1 ~]# openshift version
openshift v1.1.0.1-1-g2c6ff4b
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

I am trying to create CEPH persistent storage via rbd plugin. I have working Ceph cluster and I have followed official guide for ceph from documentation:

https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_ceph_rbd.html

I can create persistentVolume and persistentVolumeClaim without problems, but when kubernetes try to mount the volume, it report this error:

FailedMount  {kubelet node-4} Unable to mount volumes for pod "php55-rzat2_xyz": unsupported volume type
Pod  FailedSync   {kubelet node-4}   Error syncing pod, skipping: unsupported volume type

I have installed ceph, ceph-common and ceph-fuse packages on all nodes (and to be sure, I also installed it on masters).

persistentVolume is claimed by persistentVolumeClaim as expected.

pv config file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "ceph-ssd-0000-11"
spec:
  capacity:
    storage: "1Gi"
  accessModes:
    - "ReadWriteOnce"
  rbd:
    monitors:
      - "10.0.0.5:6789"
      - "10.0.0.6:6789"
      - "10.0.0.7:6789"
    pool: ssdvolumes
    image: ssd-shift-pv-0001
    user: ssdvolumes
    secretRef:          
       name: "QVxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=="
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"

I can also connect via rbd in cli without any problems.

Anyone can help me with this? I thought rbd plugin is automaticaly installed to kubernetes with openshift instalation.

score 5 · Accepted Answer · edited Feb 18 '16 at 13:56

So I figured it out. Official documentation is little bit unclear about creating ceph pv. You have to create another object "secret" with ceph secret and then just reference to it in pv configuration.

pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "ceph-ssd-0000-06"
spec:
  capacity:
    storage: "1Gi"
  accessModes:
    - "ReadWriteOnce"
  rbd:
    monitors:
      - "10.2.70.51:6789"
      - "10.2.70.81:6789"
      - "10.2.70.82:6789"
    pool: ssdvolumes
    image: ssd-shift-pv-0006
    user: ssdvolumes
    secretRef:           
       name: "ceph-secret"
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"

ceph-secret.yml

apiVersion: v1
kind: Secret
metadata:
  # coresponds to secretRef in pv.yml
  name: ceph-secret
data:
  # base64 encoded ceph secret
  key: QVxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxltZFE9PQ==

and then execute

oc create -f pv.yml
oc create -f ceph-secret.yml

Also error message : unsupported volume type is confusing, as it is reported even if the rbd volume is supported. Error caused missing secret.

The issue is reported and provably going to be solved: https://github.com/kubernetes/kubernetes/issues/18444

Edit2: Additional information about mounting ceph persistent volume: You have create image in ceph pool before mounting. Openshift (or kubernetes) wont create image by themselves.

Edit3: You must also manually mkfs.ext4 this image before using it.

OpenShift Origin and Ceph persistent volume

1 Answers1