1

I am using Ceph, uploading many files through radosgw. After, I want to delete the files. I am trying to do that in Python, like this:

bucket = conn.get_bucket(BUCKET)
for key in bucket.list():
    bucket.delete_key(key)

Afterwards, I use bucket.list() to list files in the bucket, and this says that the bucket is now empty, as I intended.

However, when I run ceph df on the mon, it shows that the OSDs still have high utilization (e.g. %RAW USED 90.91). If I continue writing (thinking that the status data just hasn't caught up with the state yet), Ceph essentially locks up (100% utilization).

What's going on?

Note: I do have these standing out in ceph status:

   health HEALTH_WARN
            3 near full osd(s)
            too many PGs per OSD (2168 > max 300)
            pool default.rgw.buckets.data has many more objects per pg than average (too few pgs?)

From what I gather online, this wouldn't cause my particular issue. But I'm new to Ceph and could be wrong.

I have one mon and 3 OSDs. This is just for testing.

UPDATE: The space does seem to be reclaimed, very slowly. It went down to 63% utilization after an hour or so.

Turtle V. Rabbit
  • 183
  • 2
  • 8

1 Answers1

1

You can use the Ceph pg calc tool. It will help you to calculate the right amount of pgs for your cluster. My opinion is, that exactly this causes your issue. You can see that you should have only 256 pgs total. Just recreate the pool (!BE CAREFUL: THIS REMOVES ALL YOUR DATA STORED IN THIS POOL!):

ceph osd pool delete {your-pool-name} {your-pool-name} --yes-i-really-really-mean-it
ceph osd pool create {your-pool-name} 256 256

It should help you.

n.shalnov
  • 11
  • 1