2

I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.

MadHatter
  • 78,442
  • 20
  • 178
  • 229
Alex Flo
  • 1,711
  • 3
  • 17
  • 23

2 Answers2

2

1) XFS now is better choice than EXT4.

2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.

poige
  • 9,171
  • 2
  • 24
  • 50
0

After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.

Alex Flo
  • 1,711
  • 3
  • 17
  • 23