2
I was copying ~3.7TB of data from one 4TB external drive with HFS+ filesystem, to another 4TB external drive with an exFAT filesystem. The new HDD got filled after only ~75% of the data could be transferred, due to, I am guessing, the allocation unit size of exFAT using up more space per file for small file sizes.
I am copying a lot (~ millions) of small files (1.5kB each). So I am trying to figure out how to do this.
Therefore I need a filesystem that fullfills the following requirements:
Block size small enough that I can fit millions of files sized 1.5kB wasting minimal space. (here exFAT has a problem)
Read/write compatible with Linux. (here HFS+ has a problem)
Able to make a 4T partition on Linux. (here ext4 has problems)
Any alternative filesystem?
UPDATE: This question was flagged as already been answered in another post (Optimizing file system for lots of small files?). However the accepted answer does not work for me:
mkfs.ext4 -b 1024 /dev/your_partition
Warning: specified blocksize 1024 is less than device physical sectorsize 4096
/dev/sdc: Cannot create filesystem with requested number of inodes while setting up superblock
hirschme
Posted 2018-08-27T20:29:01.917
Reputation: 133
1EXT4 has no problem creating 4TB partitions. – None – 2018-08-27T21:22:28.520
2
IMHO the best filesystem is ZFS that has many advantages over others, it well supported on all BSDs and Linux and even on windows (in early betas). If you want to be more compatible, then microsoft's NTFS will do the job over
– Alex – 2018-08-27T21:51:01.470ntfs-3g, it can be accessible practically on any platforms. Do not use exFAT for sure, basically it is old FAT32 system but with removed maximum limits, that doesn't support any integrity checking as well wasting space by using huge cluster size.One more for zfs – ivanivan – 2018-08-27T22:54:47.793
1
@GabrielaGarcia the problem is the tiny files, not the volume size. @hirschme the maximum volume size for ex4 is 1EiB. See filesystem for millions of small files, What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?, Optimizing file system for lots of small files?, Filesystem large number of files in a single directory. But you should clarify if your files are 150 or 1.5kB
– phuclv – 2018-08-28T05:43:17.8701
Possible duplicate of Optimizing file system for lots of small files?
– phuclv – 2018-08-28T05:45:47.370@phuclv thanks for the suggestion, I corrected the file size in the question and updated with the possible duplicate post – hirschme – 2018-08-28T14:19:02.833
@phuclv I know. I wasn't intending to answer the actual question, just correcting the part "make a 4T partition ... here ext4 has problems". It hasn't – None – 2018-08-28T15:40:40.073