2

I have a root directory 'data_0'. Under this directory are about 15,000 directories ('a', 'b', 'c', ... 'aa', 'ab' ...). Under each of theses directories there are thousands of very small files (4~10kB), something between 1,000 and 2,000 files each.

All this leads to 30 million files. I need to move these from 'data_0' to a 'data_1' folder, but without the "level 2" folders (a, b, c etc), so:

/data_0/a/1.txt --> /data_1/a_1.txt
/data_0/a/2.txt --> /data_1/a_2.txt
...
/data_0/ccc/989.txt --> /data_1/ccc_989.txt
...

How far can I go with this? Performance is unimportant here. Is there a logical limit or just a performance limit?

TessellatingHeckler
  • 5,676
  • 3
  • 25
  • 44
chesterman
  • 65
  • 1
  • 3
  • 12

2 Answers2

4

There is no logical limit to the number of files in a single directory, although there are significant performance limits as individual directories get more files (my rule-of-thumb is keep it under 1000, although I've done 10000 in certain situations). There is a limit to the total number of files on a filesystem, governed by your filesystem creation parameters (total number of inodes). There is also a hard-coded limit of 32000 subdirectories per directory (of which two entries are always taken up by . and ..).

womble
  • 95,029
  • 29
  • 173
  • 228
2

If strangers on the internet are to be trusted, there is no limit to the number of files that an ext3 folder can contain. So says the ext3-users RedHat mailing list. The 2.6 kernel supposedly allows for the theoretical allowance of "billions" of files in one directory. You may want to tweak dir_index a bit to make it run smoothly if you'll be doing in searching on the files. There are also some other side effects of massive amounts of files in one directory that you might want to read through in this StackOverflow thread.

So the answer is most likely: "Yes, but..."

Community
  • 1
Wesley
  • 32,320
  • 9
  • 80
  • 116
  • hm... very tks. i already have a directory with about 300k files and performance definitely sucks for listing files and stuff, but since the files have well defined names and i do not try to list or open anything but a single file each time, i do not suffer from this performance penalty. – chesterman Aug 19 '11 at 11:39