6

I have been through the other questions/answers regarding inode usage, mounting issues and others but none of those questions seem to apply...

df -h

/dev/sdd1 931G 100G 785G 12% /media/teradisk

df -ih

/dev/sdd1 59M 12M 47M 21% /media/teradisk

Basically, I have an EXT4 formatted drive 1TB in size, and am writing arount 12 million (12201106) files into one directory. I can't find any documentation on a files-per-directory limit for EXT4 but the filesystem reports no space left.

Oddly, I can still create new files on the drive and target folder but when doing a large cp/rsync, the calls to mkstemp and rename report no space left on device.

rsync: mkstemp "/media/teradisk/files/f.xml.No79k5" failed: No space left on device (28)

rsync: rename "/media/teradisk/files/f.xml.No79k5" -> "files/f.xml": No space left on device (28)

I know storing this many files in one directory isn't advised for a ton of reasons, but unless I can help it I don't wish to split them up.

Inode and space usage for tmpfs, the device and everything else looks fine. Any ideas of the cause?

Community
  • 1
Aiden Bell
  • 609
  • 1
  • 5
  • 15
  • 2
    Why would you need 12M files in one directory? – rvs Apr 30 '12 at 14:28
  • 2
    @rvs, sometimes you do not chose. – poige Apr 30 '12 at 14:44
  • @rvs - I am limited in what I am doing. I could in theory split the files, but I am not doing directory listing or indexing of files. If I can keep them in one dir it will make my life simpler. – Aiden Bell Apr 30 '12 at 15:40
  • So, what you've come up with? ) – poige May 14 '12 at 04:43
  • Used XFS, solved issue. – Aiden Bell Jun 15 '12 at 16:56
  • separating the files into folders shouldn't be as hard as you thought. Just split the filename in to parts and used those parts as directory name, or better, split the *hash* instead like how git is doing. See [Storing a million images in the filesystem](https://serverfault.com/q/95444/343888) – phuclv Jul 22 '18 at 02:05
  • @phuclv it wouldn't have been difficult at all, but as mentioned in the question and comments keeping the files in one directory made other things much simpler. This question was limited to issues with files in one directory. "Split them in to directories" wasn't an option nor the question. – Aiden Bell Jul 27 '18 at 12:42

6 Answers6

5

It seems you are hitting directory size limit. Directory itself is some kind of special file which contains names (+ inode numbers and probably some other metadata) of all files in it. And it can't be larger than 2G.

Anyway, it's not a good idea to have more than few thousands of files in one dir: searches by file name would be very slow and you'll have a lot of problems with standard tools like ls, rm and others.

Update:

a-ha!

http://old.nabble.com/re:The-maximum-number-of-files-under-a-folder-td16033098.html

On Mar 13, 2008 13:23 -0400, Theodore Ts'o wrote:

There is no limit to the number of files in a folder, except for the fact that the directory itself can't be bigger than 2GB, and the number of inodes that the entire filesystem has available to it. Of course, if you don't have directory indexing turned on, you may not like the performance of doing directory lookups, but that's a different story.

There is also a limit in the current ext3 htree code to be only 2 levels deep. Along with the 2GB limit you hit problems around 15M files, depending on the length of the filenames.

rvs
  • 4,027
  • 1
  • 25
  • 30
2

The XFS filesystem would be a more supportable (long-term) solution for what you're trying to do now. Large file-count directories are not a problem for XFS. Of course, fixing this at the application level would also be helpful...

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • http://serverfault.com/questions/384541/ext4-no-space-left-on-device-28-incorrect/384548#comment396127_384550 – poige Apr 30 '12 at 14:30
  • 2
    While I realize both other Filesystems cope better, and application level improvements can be made, this answer does not address the cause of the issue in ext4 other than saying "use another FS" ... Stating where ext4 and XFS differ in terms of my specific issue would have been more helpful. – Aiden Bell Apr 30 '12 at 15:28
  • 1
    While I did downvote this question (and my downvote is still valid) I did end up going with XFS. – Aiden Bell Jun 15 '12 at 16:58
  • 1
    @AidenBell ***Of course you did.*** – ewwhite Mar 06 '13 at 23:48
  • In my embedded application, XFS, sadly, is not an option for me. Problem not solved. :'( – James T Snell Sep 28 '14 at 04:50
  • @Doc which embedded application uses millions or billions of files? – phuclv Sep 08 '16 at 09:50
  • @LưuVĩnhPhúc Something I was developing. The files were map tiles. I've since taken that product in a different direction to accomplish my goal. Flat files was doable, I could rig that up now. But meh, I found another way. – James T Snell Sep 09 '16 at 20:11
  • [*One thing is that XFS doesn't perform well on millions of small sizes*](https://serverfault.com/a/505861/343888) – phuclv Jul 22 '18 at 02:34
1

Another source of ENOSPC errors when using the ext4 filesystem can be hash collisions of the directory index hash algorithm used.

This blog post talks about more details:

ext4 uses half_md4 as a default hashing-mechanism. If I interpret my google-results correctly, this uses the md4-hash algorithm, but strips it to 32 bits. This is a classical example of the birthday-paradox: A 32 bit hash means, that there are 4294967296 different hash values available, so if we are fair and assume a uniform distribution of hash values, that makes it highly unlikely to hit one specific hash. But the probability of hitting two identical hashes, given enough filenames, is much much higher. Using the formula from Wikipedia we get (with about 50K files) a probability of about 25% that a newly added file has the same hash. This is a huge probability of failure. If on the other hand we take a 64bit hash-function the probability becomes much smaller, about 0.00000000007%.

Changing the directory hash algorithm to tea should resolve the problems.

Michael Renner
  • 1,750
  • 13
  • 17
1

is ext4 absolutely needed for you? These days XFS should handle a situation like this without a hitch.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • I wouldn't say that. At least you need "delaylog" support and since it's quite a recent change it can be missing in OP's case. – poige Apr 30 '12 at 14:27
  • 1
    Have you actually tried XFS recently? With decent hardware OPs problem should be a no-problem. – Janne Pikkarainen Apr 30 '12 at 14:32
  • This is not OP question was about. ext4 is pretty good choice and I don't think XFS will really help to deal with millions of files in one directory. – rvs Apr 30 '12 at 14:33
  • 1
    @JannePikkarainen, I actually did (and not once). That's why I wrote a note saying that it's actually faster to copy content of XFS than remove it ( http://poige.livejournal.com/471842.html ) – poige Apr 30 '12 at 14:39
  • 2
    Again, my question isn't "Ext4 or XFS for millions of files in one directory" but "I have issue X on ext4, what could be the direct cause."; "You are using EXT4" isn't an answer to "what is the cause?" – Aiden Bell Apr 30 '12 at 15:29
1

I had this problem. My solution was:

mkfs.ext4 -i 1024 -b 1024 /dev/blah

Dan Brough
  • 111
  • 2
  • This would require him to reformat the partition, no? Not sure if this really helps him, he may not be able to do this. – slm Feb 20 '13 at 14:17
  • Ext4 creates 256 byte i-nodes by default. Since your i-nodes are only keeping track of 1024 bytes, per the `-i 1024` argument, 20% of your disk will be dedicated to i-nodes. If your filesystem is full of files that take the minimum 1 block of storage, this makes sense. This is extremely uncommon however, and most people will not want to use these extreme settings. – Chris S Feb 20 '13 at 14:34
  • yup, I was testing out ext4 with the gentoo portage directory. To reproduce this error just try to rsync the gentoo portage directory into a loopback file created with the default mkfs.ext4 options. – Dan Brough Feb 20 '13 at 14:46
-4

It seems you're running out of i-nodes. Show output of df -iht ext4.

I also have had a problem with removing a directory on EXT4 containing ~ 1 million of files (linux kernel 3.0, IIRC). What's kernel version in your case?

Finally, I'd suggest using Reiser3 — it doesn't have format time i-node limits, and in aforementioned case seemed to have solved the problem as well.

UPD.: For those wondering if Reiser3 being supported on not:

cd linux-stable/fs/reiserfs && git log --pretty='format:%aD %s' . | head -n20

Tue, 10 Jan 2012 15:110:11 -0800 reiserfs: don't lock root inode searching
Tue, 10 Jan 2012 15:11:09 -0800 reiserfs: don't lock journal_init()
Tue, 10 Jan 2012 15:11:07 -0800 reiserfs: delay reiserfs lock until journal initialization
Tue, 10 Jan 2012 15:11:05 -0800 reiserfs: delete comments referring to the BKL
Mon, 9 Jan 2012 12:51:21 -0800 Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs
Wed, 21 Dec 2011 21:18:43 +0100 reiserfs: Force inode evictions before umount to avoid crash
Wed, 21 Dec 2011 17:35:34 +0100 reiserfs: Fix quota mount option parsing
Wed, 21 Dec 2011 20:17:10 +0100 reiserfs: Properly display mount options in /proc/mounts
Wed, 7 Dec 2011 18:16:57 -0500 vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb
Tue, 26 Jul 2011 02:50:53 -0400 reiserfs: propagate umode_t
Tue, 26 Jul 2011 01:52:52 -0400 switch ->mknod() to umode_t
Tue, 26 Jul 2011 01:42:34 -0400 switch ->create() to umode_t
Tue, 26 Jul 2011 01:41:39 -0400 switch vfs_mkdir() and ->mkdir() to umode_t
Mon, 12 Dec 2011 15:51:45 -0500 vfs: fix the stupidity with i_dentry in inode destructors
Fri, 9 Dec 2011 08:06:57 -0500 vfs: mnt_drop_write_file()
Wed, 23 Nov 2011 11:57:51 -0500 switch a bunch of places to mnt_want_write_file()
Fri, 28 Oct 2011 14:13:29 +0200 filesystems: add set_nlink()
Fri, 28 Oct 2011 14:13:28 +0200 filesystems: add missing nlink wrappers
Tue, 25 Oct 2011 12:11:02 +0200 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
Thu, 15 Sep 2011 15:08:05 +0200 Merge branch 'master' into for-next
poige
  • 9,171
  • 2
  • 24
  • 50
  • 4
    Has the Reiser community sufficiently recovered from the whole "lead dude murdered his wife" thing to really be considered a production alternative to EXT4? – MDMarra Apr 30 '12 at 14:19
  • @MDMarra, don't be silly. I'm talking bout Reiser3, not Reiser4. – poige Apr 30 '12 at 14:25
  • 5
    @poige Reiser4 is pretty advanced, having learnt lessons from previous incarnations. Version 4 now burns down the house and dumps the car in the lake after homocides. – Dan Apr 30 '12 at 14:27
  • 1
    @Dan, have you tried? I at least did. It lacks O_DIRECT, no resizes, and alas not too stable. Hence, the only good choice of Reiser FS is version 3. – poige Apr 30 '12 at 14:29
  • @poige My comment was more about the ReiserFS as a whole. With the instability in v4 and absence of the lead developer, why push someone toward it when its future seems bleak, especially when it's not the only alternative? – MDMarra Apr 30 '12 at 14:35
  • @MDMarra, I can only repeat my advice. See logs (special for you). – poige Apr 30 '12 at 14:36
  • 2
    @poige I was joking, but on a serious note, the company that is meant to develop Reiser is formally suspended with little hope of returning. It doesn't seem like the "way forward" to now begin using it in production. – Dan Apr 30 '12 at 14:36
  • @poige I never doubted that it was still being developed, I only questioned the **quality** of that development. I have no doubt that it will solve the technical issue at hand, but it is *not* the only way to solve it. My point was simply that other filesystems might be more appropriate in the long-term based on the political and leadership problems surrounding the Reiser stuff in general. I think we're going to have to agree to disagree ;) – MDMarra Apr 30 '12 at 14:39
  • @Dan, I knew you was. I just don't give a crap about "theoretic" opinions since I have practice. And it is a good one. – poige Apr 30 '12 at 14:41
  • @poige The comments that I've written on your answer are *exactly* what the comment field is for. If you can't deal with some detraction, then perhaps you don't belong on this site. – MDMarra Apr 30 '12 at 14:44
  • @poige While this isn't a discussion site it's still encouraged to discuss answers, especially controversial ones. Your answer may not be outright wrong, hell maybe it *is* the best one, but there's still room for discussion and other opinion and it's entirely fair for the original poster to see that. Phrases like "Keep your points in your own answers" are idiotic - this isn't your private, secure answer. We can even edit your original answer, if we wanted. – Dan Apr 30 '12 at 14:45
  • @MDMarra, you seem to belong to some non-technical sites, asking about "lead dude murdered his wife". It has nothing to do with code quality and its support. – poige Apr 30 '12 at 14:46
  • @poige It may have been open source, but open even (good) open source projects need a clear lead. That's clear by the fact that Reiser was developed by a company headed up by Hans Reiser. It's fair to ask questions as to the future of the filesystem now all that infrastructure has gone – Dan Apr 30 '12 at 14:48
  • @Dan, it's a theory again. Feel free to fantasize further… – poige Apr 30 '12 at 14:50