3
1
I would like to know what is the most efficient way of concurrently accessing thousands of files of a similar size in a modern Linux cluster of computers.
I am carrying an indexing operation in each of these files, so the 4 index files, about 5-10x smaller than the data file, are produced next to the file to index.
Right now I am using a hierarchy of directories from ./00/00/00
to ./99/99/99
and I place one file at the end of each directory,
like ./00/00/00/file000000.ext
to ./99/99/99/file999999.ext
.
It seems to work better than having thousands of files in the same directory but I would like to know if there is a better way of laying out the files to improve access.
Is there any reference to this kind of things? – kawing-chiu – 2017-11-22T08:51:08.993
@kawing-chiu, if you search the e2fsprogs mailing list archive, you should be able to find a thread I was in discussing it. I even wrote a little python script to scan a directory and report the amount of correlation from directory entry -> inode number, and inode number -> starting block. I found my maildir that contained many thousands of files had very poor name -> inode number correlation. – psusi – 2017-11-22T22:07:34.207
@psusi Thanks for the information. From my test, iterating by inode has a 30% speed increase indeed. – kawing-chiu – 2017-11-23T02:19:23.443
@kawing-chiu, you might also be interested in http://launchpad.net/e2defrag. It's a very old ext2 offline defragmenter that can pack all files in perfect order, or even pack all files used during boot at the start of the drive to speed up boot times, when used in conjunction with ureadahead or e4rat.
– psusi – 2017-11-24T01:12:39.350how do I sort the directory listing by inode number? – 719016 – 2012-07-04T07:49:15.213
@130490868091234, readdir() returns the name, and inode number for each entry. Sort the array of entries, in order from lowest to highest inode number. – psusi – 2012-07-05T01:25:07.047