4

We have a weird behaviour that we can not move files into a certain directory anymore. We get 

lstat("NewBatches/R910140805849312.dat", {st_mode=S_IFREG|0644, st_size=2850, ...}) = 0
lstat("Imported/R910140805849312.dat", 0x7fff10424b90) = -1 ENOENT (No such file or directory)
rename("NewBatches/R910140805849312.dat", "Imported/R910140805849312.dat") = -1 ENOSPC (No space left on device)

But we can copy the file into the folder. There is plenty of diskpace left and also inodes. And we can not move the file just in that Imported subdirectory. All others work within the same EXT3 filesystem.

I am a bit puzzled

# tune2fs -l /dev/mapper/vgdmscsp-lvmaspdoc
tune2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          b4215e24-2285-46de-8398-f41bc3174b8e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              33382400
Block count:              52428800
Reserved block count:     2619904
Free blocks:              5432592
Free inodes:              17432375
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      176
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         5216
Inode blocks per group:   652
Filesystem created:       Thu Oct  6 11:19:53 2011
Last mount time:          Sat Jul 12 09:26:56 2014
Last write time:          Tue Aug  5 00:04:31 2014
Mount count:              40
Maximum mount count:      -1
Last checked:             Thu Oct  6 11:19:53 2011
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      b975b5a1-72ad-44a4-8c53-622f7ba71e25
Journal backup:           inode blocks
Chris
  • 597
  • 1
  • 6
  • 18
  • 1
    Did you try to run `fsck` on the file system? Did it report any problem it could not solve? – Huygens Aug 05 '14 at 13:35
  • not yet. As it is in production. Will have to wait another 2 weeks for that. But we planned it to do the fsck. – Chris Aug 05 '14 at 13:41
  • 1
    It seems that you are using ext3 on top of LVM. You can do a snapshot of the volume and perform a fsck on it (instead of the mounted system). So you can do it without downtime. See this article and provided links: http://prefetch.net/blog/index.php/2011/10/15/checking-ext3-file-system-consistency-on-production-systems/ – Huygens Aug 05 '14 at 15:41
  • 1
    Huygens. This is absolutely awesome. I did not think of this feature! Thanks for that excellent tip! – Chris Aug 06 '14 at 07:33
  • What happens if you try to do `ln` of the file into that directory and `rm` of the file from the old one? Hard link obviously. Also can you provide the `stat` outpuf of the source directory, target directory and the file? – Matthew Ife Aug 09 '14 at 18:27

1 Answers1

1

Your free block count isn't very far off from the reserved count.

Block count:              52428800
Reserved block count:     2619904
Free blocks:              5432592

At creation time the ext3 filesystem reserves a percentage of blocks for use by the root user, which is 5% by default. This allows root owned processes to continue writing to disk while locking out userspace, which one hopes is the source of the disk bloat.

I suspect you were running into this problem as an unprivileged user while the number of free blocks was below the reserved count. The cp would have succeeded if you were intervening as root. If you can confirm that the problem is gone now and the current number of free blocks is above the reserved count, that is the most likely cause.

Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • Unless the directory will grow I dont necessarily see how a `mv` would acquire more blocks (assuming its staying on the same filesystem). – Matthew Ife Aug 09 '14 at 18:24
  • @Matthew You're assuming the same filesystem. Given the symptom set, I'm assuming they aren't. This answer will be useful to someone and I'm not going to sweat it. – Andrew B Aug 09 '14 at 18:29
  • I've asked if the user can provide the stat results of all relevent inodes, so we'll know for certain what the layout is then. – Matthew Ife Aug 09 '14 at 18:31
  • @AndrewB A cross device `rename` fails with `EXDEV` not `ENOSPC`. If the file system supported directory quotas, it could make sense for renames inside the same file system to sometimes fail with `ENOSPC`. But I don't think such a feature exists in `ext3`. – kasperd Aug 09 '14 at 19:23