0

I have a production server with lustre file system in it. It has 2 OSTs and one MDT. Now i my MDT is full. I can't delete my old files.

Here a snapshot of my lustre file system.

$ lfs df -i
UUID                      Inodes       IUsed       IFree IUse% Mounted on
data-MDT0000_UUID       28002745    28002745           0 100% /mnt/data[MDT:0]
data-OST0000_UUID       78818692    13987183    64831509  18% /mnt/data[OST:0]
data-OST0001_UUID       78823552    13987089    64836463  18% /mnt/data[OST:1]

filesystem_summary:     28002745    28002745           0  18% /mnt/data

We can't buy new MDT in the near future, so our best hope is to delete some old files.

But when we try to delete the files, we always get these error

rm -rf *
rm: cannot remove ‘/foo/bar/file1.png’: No space left on device
rm: cannot remove ‘/foo/bar/file2.png’: No space left on device

We have tried these method but it fails, "rm: can not remove xxx: No Space left on device" on BTRFS

Looks like lustre have different metadata from BTRFS.

How can we delete our files?

Edit We are adding two hard drive (sda sdc) to existing MDS server, and creating new mds with it. And tried to attach it to our existing lustre file system.

mkfs.lustre --mdt \
   --fsname data \
   --index 1 \
   --mgsnode 20.20.2.1@o2ib \
   --servicenode 20.20.2.1@o2ib \
   --backfstype=zfs \
   mdt1pool/mdt1 \
   mirror sda sdc

mkdir -p /lustre/mdt1 && mount -t lustre mdt1pool/mdt1 /lustre/mdt1

But it returns this error.

mount.lustre: mount mdt1pool/mdt1 at /lustre/mdt1 failed: No space left on device
luthfi.imanal
  • 1
  • 2

1 Answers1

0

Can you try to delete one file first? '*' is manipulated differently in bash, and since no space available it can't create tmp file too, Hope it will help you.

asktyagi
  • 2,401
  • 1
  • 5
  • 19
  • i've tried to delete one file with rm or unlink. But no luck. It seems my best bet is adding storage directly to my zpool. – luthfi.imanal May 08 '19 at 07:40