I have a ceph cluster with 2 nodes and 3 osd's each. Each osd is on one partion on a 8TB disk. The server is limited in amount of disk so I even have system on the same disks.
On install I used prepare and activate on the partion and everything got started up fine.
ceph-deploy osd prepare burk11:/dev/sda2 burk11:/dev/sdb2 burk11:/dev/sdd2
ceph-deploy osd activate burk12:/dev/sda2 burk12:/dev/sdb2 burk12:/dev/sdd2
It works fine. But after little testing I was planing of use it, start copied data to it and it got full very quickly and when looking at the size its only 60GB and I can't figure out why.
ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
61440M 19593M 41846M 68.11
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
cinder-backup 1 0 0 1851M 0
cinder-volumes 2 0 0 1851M 0
glance-images 3 0 0 1851M 0
.rgw.root 4 1113 0 1851M 4
default.rgw.control 5 0 0 1851M 8
default.rgw.meta 6 1661 0 1851M 10
default.rgw.log 7 0 0 1851M 207
default.rgw.buckets.index 8 0 0 1851M 2
default.rgw.buckets.data 9 14109M 88.40 1851M 38929
But the disk are almost empty
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdc2 7.3T 4.0G 7.3T 1% /var/lib/ceph/osd/ceph-4
/dev/sdd2 7.3T 8.4G 7.3T 1% /var/lib/ceph/osd/ceph-5
/dev/sda2 7.3T 8.2G 7.3T 1% /var/lib/ceph/osd/ceph-3
root@burk11:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.05878 root default
-3 0.02939 host burk11
0 hdd 0.00980 osd.0 up 1.00000 1.00000
1 hdd 0.00980 osd.1 up 0.95001 1.00000
2 hdd 0.00980 osd.2 up 1.00000 1.00000
-5 0.02939 host burk12
3 hdd 0.00980 osd.3 up 1.00000 1.00000
4 hdd 0.00980 osd.4 up 1.00000 1.00000
5 hdd 0.00980 osd.5 up 0.95001 1.00000
root@burk11:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME
-1 0.05878 - 966G 954G 11905M 98.80 1.00 - root default
-3 0.02939 - 30720M 21579M 9140M 70.24 0.71 - host burk11
0 hdd 0.00980 1.00000 10240M 5650M 4589M 55.18 0.56 172 osd.0
1 hdd 0.00980 0.95001 10240M 8380M 1859M 81.84 0.83 139 osd.1
2 hdd 0.00980 1.00000 10240M 7548M 2691M 73.72 0.75 153 osd.2
-5 0.02939 - 936G 933G 2764M 99.71 1.01 - host burk12
3 hdd 0.00980 1.00000 10240M 9589M 650M 93.65 0.95 163 osd.3
4 hdd 0.00980 1.00000 10240M 9309M 930M 90.91 0.92 144 osd.4
5 hdd 0.00980 0.95001 916G 914G 1183M 99.87 1.01 157 osd.5
TOTAL 966G 954G 11905M 98.80
MIN/MAX VAR: 0.56/1.01 STDDEV: 22.15
(And no, I don't use 914G on OSD.5 I have tried to expand the block file with dd, I did get the osd bigger but even if it's not using more data I got it full, should have a usage around 9-10G)
So how can I make ceph use the full partion?
