1
We are using CEPH as backend for our OpenStack installation.
Since two weeks, we see that we are nearly (and now) full on our volumes:
$ ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
84887G 34052G 50834G 59.88
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
.rgw.root 1 1113 100.00 0 4
images 2 229G 100.00 0 38486
volumes 3 15668G 100.00 0 4021432
backups 4 19 100.00 0 2
vms 5 841G 100.00 0 218124
default.rgw.control 6 0 0 0 8
default.rgw.meta 7 0 0 0 0
default.rgw.log 8 0 0 0 207
volumes-SSD 9 235G 35.15 434G 60442
I see that we cannot use all the disk as we have servers with few disks and servers with many disks and so Ceph cannot distribute as evenly as hoped (we have a replica of 3).
3 times Sum of POOLS USED is roughly the same as RAW USED (because of replica of 3).
Now I try to dig into the huge volumes pool to see which one(s) are the greedy:
rbd du -p volumes
NAME PROVISIONED USED
01d01dc5-3a6f-4618-85f2-b9eaf5b09c16 20480M 2624M
0211d80a-2b6b-4723-a6b6-64c6ddfd6fd3 61440M 11332M
035c8313-cc4f-4464-aa11-3980ba565210 61440M 10708M
0362cad8-4ecb-4861-a172-cb3ea5c8e669 40960M 3320M
049ad9c7-f19b-4316-a19a-9c4b70ad67f5 81920M 16948M
06760df4-1832-4203-89d1-2b1cd102a1c4 5120M 1560M
07a6e824-6a32-478c-b216-c869b3aa14b7 61440M 34596M
08ec14e2-3128-43a1-8b94-dd129aedaff6 61440M 31700M
0b63ef53-d9a8-4d8f-80c8-7e991ba9f56b 61440M 8116M
1128e783-1781-49cd-9094-d503c02d1d4e 61440M 17328M
1b1cc7e8-b696-448d-a7b6-9171b45f2ddd 1024M 57344k
1cf1d860-6f31-4914-ba47-081e6c93dae6 81920M 15372M
1f48e6bc-8c14-4032-a054-e67b31fb4a1d 61440M 28796M
1f6c62f2-d475-420a-b92d-341218b2c7c5 5120M 1104M
22e087ec-7bff-47a7-9a7e-cb641ff5f1ed 61440M 8204M
23a6a911-14c8-4110-b539-74c0ce243b72 61440M 8092M
25c292e5-87cd-41c5-baab-566c2dda2a15 61440M 6556M
28fd8021-4c9e-4879-ad5c-b1532e6520bb 40960M 1660M
2f8c06d1-0055-4ee8-ac35-15febac7f8d6 5120M 1088M
32180db8-7eff-42f4-9f43-3c43a7ae913c 40960M 4356M
333df83d-ac47-4562-b8ff-c0e9d56a95a1 61440M 11696M
3465e12c-f52a-4ca6-835b-414b154ba419 81920M 12644M
35d6422c-9ba1-465c-b608-4ca4ecfe8b4f 1024M 61440k
36e4b104-6b98-4167-b7a1-036e597493f3 61440M 9304M
37520afd-2668-47fa-a45c-e6600106769f 1024M 57344k
3b218839-eb0b-43dd-9c65-f2b66586e421 61440M 27716M
3d5cf531-5b07-4b06-bad5-82a31a794a73 61440M 34696M
3fcccb51-bc38-4355-ab50-cc35ed73920e 61440M 11580M
418a5a71-1413-4504-8064-bb2085409a41 61440M 12016M
4196ac78-6a1f-4413-bced-66d041fc2e1f 40960M 40904M
41a1d494-8de8-47d2-83d0-c39b3d6f6b74 81920M 1276M
422d3105-eb07-47e4-ab4b-3ccea7b2e55d 61440M 7172M
4371ca6c-3c02-4b17-89d5-a2f327a5cd9a 40960M 0
43bd81da-7b51-47e4-899e-9fc1dc8f1ab2 5120M 1104M
4431ff9e-d710-46df-b454-5dc80ca21ade 40960M 40960M
45b0759f-fe7c-40d5-9f39-360806a60c06 61440M 11944M
473ad663-8d1a-4f67-b7c7-e7a5b739729a 40960M 19596M
4bbb9b2e-e298-49a8-978e-4f3986395b9c 61440M 12988M
5173177a-322e-4d56-ac4f-b04acd190285 61440M 13068M
56899172-13b8-4d87-b372-c7237ecb8b23 1024M 57344k
56fe0f14-e75d-44ac-82bc-d2c8c72195c5 61440M 40736M
58f28005-081a-4046-b723-7fd69a2cf84f 1024M 61440k
644f97db-6cd4-4855-83c2-95f914be69ea 500G 378G
6a12916d-f707-41aa-8b29-67abde7f4845 61440M 9304M
6b7e2d3d-2354-456b-8e69-b0d5d08acb45 61440M 11328M
6ce54dae-162e-415f-a27d-cfb900c9b30e 81920M 33492M
6d03ad58-7db8-48b1-96f3-70d408579a8a 61440M 41556M
6d076e6c-2cae-4796-8a4f-42920c94ddfc 61440M 12052M
6e96a285-5191-4fdc-a181-e98df04c2722 61440M 13920M
7065dacc-f4af-4a4c-8429-82a3c06d9bf8 61440M 12052M
73d135f1-83a2-45fd-9856-7b16dadd7ac9 102400M 19940M
73d198f1-bee2-416d-9826-81e7340393c1 102400M 43908M
746d2101-84ab-4afb-93bf-23000b1c24f7 5120M 2052M
7c2f605d-dd85-4426-a843-287cab2eb233 1024M 57344k
7f935336-25c7-449b-8e31-b6f9cf8de1fe 61440M 7684M
82cf1bae-83ba-4183-9fd5-be6fe5fdbca0 61440M 8104M
8666a7c9-123b-4ebd-aafe-a262e542069d 81920M 11484M
87876fc1-9be4-44fd-8b8e-b9a413b4ea5e 10240M 69632k
8a108ce7-ecf1-4d39-a65a-74f72f30ef36 5120M 996M
8ba11bdb-85a0-46ac-8358-1f0f0cda4900 81920M 1192M
8c10cc4b-9ceb-4a22-930c-17d88a4577e7 61440M 9808M
8cad9cc2-80e5-42f1-a146-b93fe7fb544a 5120M 1112M
8d11685e-2cf4-477d-8d27-b595e73dade6 61440M 8072M
91b3cd3c-e26b-4b9c-84dc-26bbf2c1ef5f 61440M 37808M
96441ac6-cd7f-4a3e-b42f-97bcb0c1e2db 1024M 57344k
9919e429-ea9e-4447-b0e5-506c9453c1a7 102400M 0
99e71d07-2da4-4cf2-b12d-e1305be15659 61440M 12956M
9ed969a8-7b42-486f-8de6-1205f393c8ab 61440M 11064M
a139ea0f-b7bd-4148-a8b7-80f77497838e 40960M 0
a2b46610-1919-4ac0-a8e0-e76b7ce1221f 5120M 996M
a41cce51-99c4-4ccd-948c-ceefd2bbbc1a 81920M 27236M
a4d51863-d4b3-487d-87eb-37bcb71719c7 61440M 32988M
a601b2b6-b413-423d-9210-4afae5115d1f 1024M 57344k
a803d575-6539-4f9f-8ce8-d2058602a6bf 40960M 24660M
a989949e-9377-4929-b4cf-961779c76ca8 81920M 19156M
a9e37a84-4ae5-47b9-9c81-34a34a411c48 81920M 25104M
aa2e258a-8d49-4cbe-ab00-545b3c564b83 1024M 61440k
aaff1b72-c59a-478a-85be-da412630765b 10240M 69632k
afd36bea-8dbb-407e-a14a-0db6f9189b4b 2048M 0
b0ae7c4e-1ca4-4015-a461-d4d43feafff5 81920M 14444M
b4109743-3e11-4938-9e99-b3c86161a7dd 81920M 13880M
b421adae-53e5-48f9-9b9d-d7ecaf6de8b7 61440M 11092M
b57d1fe9-7604-4945-b7c3-39c6221388bc 61440M 8384M
bb72bc84-e521-4c92-93a2-2f5646ff7f38 61440M 27564M
bc802e71-a5b0-4ba7-aa15-237685bd3dd8 40960M 0
becca2bd-4a01-4c89-889c-d28c803ff0bd 10240M 69632k
c399ab72-f783-4b4d-b484-b50ed59c4509 5120M 1044M
c3b010a5-3861-40c9-bb93-92d18ac6d6b2 61440M 9140M
c47666fc-952f-4ee3-ab81-ddfd235be4af 1024M 57344k
c4e4867a-83b2-4df4-acaa-f5ef25889d32 61440M 15668M
c65bf903-7e2c-4f05-91bf-fe995075b017 61440M 9560M
cc6cdd4d-b515-4dc6-ab73-636911953e0d 5120M 1636M
d268485d-b7fb-49b5-9ff6-e889e86938c0 61440M 9696M
d5a13ee4-65c9-440d-8656-96820d99ec78 61440M 36332M
d88349c6-ae0f-4229-8324-9cadca9178d3 61440M 11724M
d8f38e2e-f84e-4f7b-acde-043b1e7bd492 5120M 1072M
d906b45e-b596-4501-92d6-c859d481f20e 61440M 11100M
dadc9e91-1c23-443b-bd0e-fa58cee61863 40960M 40960M
ddc18e8c-11cf-4d5b-a632-81f93afc8bc1 40960M 0
e051dd8d-e91b-4f38-85af-c83ca0c3da79 40960M 31756M
e16d18fb-c79d-49db-8145-4680cf0924c6 61440M 3376M
e2631386-ca49-4be0-b650-3e9cff9f4a20 81920M 13132M
e733a6e7-373f-4109-9330-531eee5b029c 1024M 57344k
ea386235-ee6c-4061-87c0-afdf9a70aecd 81920M 13580M
f0f43900-9db5-453c-b0e9-ed7668b66bf4 61440M 12580M
f49a5c36-992d-4cb6-b95c-20288e6f84dd 5120M 1544M
f7998111-8a8e-4c9b-a0e7-d75de7a7571d 61440M 35464M
f9de8283-7c10-459b-a42f-6858763e4493 1024M 57344k
faeb9623-4911-4c3d-ad01-5c4c2add3352 40960M 32924M
<TOTAL> 5484G 1678G
Here du tells me total usage of 1678G and 5484G as provisionned.
So how can I have 1678G of usage on one hand and 15668G on the other hand?
Sylvain Desbureaux
Posted 2019-08-30T12:49:55.533
Reputation: 11