I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.
Asked
Active
Viewed 651 times
2
2 Answers
2
1) XFS now is better choice than EXT4.
2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.
0
After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.
Alex Flo
- 1,711
- 3
- 17
- 23
-
What tool did you use to move files from ramdisk to HDD? `mv` or `rsync` or ? – Mark Lakata Mar 18 '21 at 23:50