Is there a way to make ext-filesystems use less space for themselves in Linux?

I have a bunch of external and internal HDDs that I use on a Linux system. I only have Linux systems, so using a Linux file-system would only make sense, right? However I'm currently using NTFS everywhere, because it gives me the most usable space out of HDDs.

I would like to switch to Linux file-systems now though, mostly because of permissions and compability (e.g. I can't get my LUKS encrypted NTFS partition to resize under Linux, keeps telling me to chkdsk under Windows).

However when I formatted those HDDs I tried out a bunch of different filesystems and every Linux filesystem, even ext2 which as far as I know has no journaling, used a lot of space for itself. I don't recall exact values, but it was over 100GB that NTFS got me more on a 2TB HDD, which is a lot.

So my question is: Is there a way to make ext-filesystems use less space for themselves? Or is there another filesystem (I've tried ext2, ext3, ext4, NTFS and vfat - None of them came even close to the usable space NTFS offered me) with perfect Linux support and great usable space?

I'd love to hear about how and why filesystems (especially ext2 which has no journaling) use that much more space than NTFS and I don't know where else to ask. I'd mostly prefer a way to use ext4 without journaling and anything else that uses up this much space, if that's possible.

confetti

Posted 2018-08-05T18:36:58.197

Reputation: 1 625

Have you seen this thread?

– JakeGould – 2018-08-05T18:42:24.103

4I have, and it explained what uses up the extra space but the difference between NTFS and ext is MUCH bigger than between reiserfs and ext, and I'm wondering if there is any way to make it smaller. For example on a 1TB HDD I'm able to use 989GB with NTFS. ext4 would give me around 909GB. – confetti – 2018-08-05T18:48:22.847

Fair enough. Decent question and the answer is enlightening too. – JakeGould – 2018-08-05T22:33:37.750

3how do you actually measure what space is available? this is important because depending on what values you look at you may or may not see the effect of the 5% reservation for example as is stated in the linked question – eMBee – 2018-08-06T04:04:20.537

I used to check the "Free space" properties in the file manager, df and gnome-system-monitor. However the latter seems to have another column called free that actually shows the free space including the 5% which I just found out about though. – confetti – 2018-08-06T04:31:06.573

2Keep in mind that journaling on file systems such as ext3 and ext4 is a good thing. It's pretty easy to lose power to an external drive or unplug it by accident if it's a USB, When that happens, it's often no big deal because it heals itself using the journal when it starts back up. Without that safety net, things would be much worse. It's not just a case of more is better. – Joe – 2018-08-06T20:18:33.223

That still applies for LUKS encrypted devices, right? I'm assuming the journal recovery/fsck will be performed on mount of the partition inside the LUKS-container, correct? @harrymc's answer gives me enough space already to use ext4 and keep the journaling. – confetti – 2018-08-06T20:24:22.267

Answers

By default, ext2 and its successors reserve 5% of the filesystem for use by the root user. This reduces fragmentation, and makes it less likely that the administrator or any root-owned daemons will be left with no space to work in.

These reserved blocks prevent programs not running as root from filling your disk. Whether these considerations justify the loss of capacity depends on what the filesystem is used for.

The 5% amount was set in the 1980s when disks were much smaller, but was just left as-is. Nowadays 1% is probably enough for system stability.

The reservation can be changed using the -m option of the tune2fs command:

tune2fs -m 0 /dev/sda1

This will set the reserved blocks percentage to 0% (0 blocks).

To get the current value (among others), use the command :

tune2fs -l <device>

harrymc

Posted 2018-08-05T18:36:58.197

Reputation: 306 093

10This would explain the immense difference in usable space perfectly (as 5% of 2TB are 100GB). The disks won't be used for anything as root or system-file related, so I think it would be save to disable this. I got a question though: How do root-owned programs know there is more free space than non-root programs? Running df as non-root vs. root shows no difference. – confetti – 2018-08-05T18:50:36.600

12@confetti: Because the VFS doesn't reject their attempts to write to the disk with an error (until the volume is actually full, of course). – Ignacio Vazquez-Abrams – 2018-08-05T18:57:04.013

Linux handles this internally by the process uid or its group. I suppose that this is done in the disk driver when allocating space, on a much lower level than the one df operates in. – harrymc – 2018-08-05T18:58:51.170

So there is no way, as the root user, to tell how much free space I actually have? What happens when root actually makes use of these 5%? Is there no way to "track" this? – confetti – 2018-08-05T18:59:29.983

gnome-system-monitor has a column called free in its file system monitor. That shows me 26.5GB for a 59GB ext4 SSD partition at the moment, while available reports 23.5GB. I assume this is the 5% difference? – confetti – 2018-08-05T19:01:49.777

1tune2fs -l <device> should give this value among others. The 5% amount was set in the 1980s when disks were much smaller, but was just left as-is. Nowadays 1% is probably enough for system stability. – harrymc – 2018-08-05T19:04:17.870

7XFS reserves the smaller of 5% or 8192 blocks (32 MiB), so the reserved amount is generally tiny compared to the size of the filesystem. – Michael Hampton – 2018-08-05T20:41:50.647

@confetti: what happens when the root user uses the 5%? 5% reserved means that a non-root user is only allowed to write as long as there is 5% of space free. in other words the 5% will never be touched until all other space is used up. – eMBee – 2018-08-06T04:07:56.957

5Thank you very much everyone for the explanations. This helped me understand greatly. My disk used to fill up entirely to its last byte before, yet my system did not fail completely, now I understand why. – confetti – 2018-08-06T04:34:09.023

"yet my system did not fail completely". But that's a good thing, and shouldn't be changed. – RonJohn – 2018-08-06T12:00:47.923

Ah yes. I had a system that was at 104% after install. Only root could write to the disk. – Joshua – 2018-08-06T16:38:02.170

@RonJohn Correct, I'm talking about external USB drives in my question that only store non-system files. – confetti – 2018-08-06T18:25:52.200

if the data you intend to store on it is compressible, btrfs mounted with compress=zstd (or compress-force=zstd) would probbaly use significantly less disk space than ext*

this will make btrfs transparently compress your data before writing it to disk, and transparently decompress it when reading it back. also, ext4 pre-allocate all inodes at filesystem creation, btrfs creates them as needed, i guess that might save some space too.

hanshenrik

Posted 2018-08-05T18:36:58.197

Reputation: 447

1Do you mind adding more information to this answer? (How it works, what it does, maybe a reference, ...) – confetti – 2018-08-06T08:04:24.480

@confetti like this? https://patchwork.kernel.org/patch/9817875/

– hanshenrik – 2018-08-06T08:06:11.227

I really like this idea, but more information about how this would impact speed and performance and such would be nice. – confetti – 2018-08-06T19:53:26.237

2@confetti, since you're using a hard drive, it'll probably improve performance. CPUs are so much faster than hard drives that the slow part of disk access is getting the data on and off the disk; the time spent compressing or decompressing won't be noticeable. – Mark – 2018-08-07T01:55:47.213

2On the other hand, most types of large files nowadays (e.g. images, audio, video, even most rich text document formats) tend to be already compressed, and generally do not benefit from additional compression. At least not of the simple general-purpose kind performed at the filesystem level. – Ilmari Karonen – 2018-08-07T10:33:37.393

@IlmariKaronen true, but btrfs automatically detect those files, and skips compressing them (unless the btrfs heuristics - which is very simple btw - fails, also the heuristics can be disabled by using compress-force instead of compress, but that's an unusual configuration ) ... there's also html, javascript, and css.. and btrfs compression works wonderfully on my email server, which stores emails in a html-like format – hanshenrik – 2018-08-07T12:49:18.883

BTRFS is a reasonable replacement for ext4 in many cases, but do note that it requires periodic maintenance and tends to be slightly slower throughput-wise depending on what you're doing. Of course the ability to modify it online and the checksumming RAID abilities to prevent data corruption generally make up for it in cases where you're stuffing the disks full and the data only changes slowly. – Perkins – 2018-08-07T17:57:08.570

Another point which has not been talked about yet is the number of inodes you reserve on your file system.

Per default, mkfs creates a number of inodes which should make it possible to put a whole lot of very small files into your file system. If you know that the files will be very big and you will only put a small number of files on the FS, you can reduce the number of inodes.

Take care! This number (resp. the ratio between space and number of inodes) can only be set at the file system creation time. Even when extending the FS, the ratio remains the same.

glglgl

Posted 2018-08-05T18:36:58.197

Reputation: 1 327

Alternatively, if you know you're going to store lots of really tiny files, you can increase the number of inodes and decrease the block size so that you don't waste as much space. (Each file must consume a minimum of one block, even if it's 1 byte. use ls -ls to compare the size vs what's used on the disk.) – Perkins – 2018-08-07T17:59:10.627

@Perkins You are right, but I think this holds only for VERY tiny files: the default block size is 4kiB (IIRC), the minimum one is 1 kiB. So not so very much to win, except your disk is really full of those files. But nevertheless, I might dive into this tomorrow. – glglgl – 2018-08-07T20:49:41.063

2or alternatively use btrfs, which creates inodes as needed. whereas ext4's inodes are allocated at filesystem creation time and cannot be resized after creation, with a hard limit of 4 billion, btrfs's inodes are dynamically created as needed, and the hard limit is 2^64, around 18.4 quintillion, which is around 4.6 billion times higher than the hard limit of a maxed ext4 :p – hanshenrik – 2018-08-07T22:53:01.813

Reminds me of setting up a usenet spool. ext4 will store tiny (limit is somewhere from 60-160 bytes depending on lots of things) within the inode itself. – mr.spuratic – 2018-08-08T08:43:53.533

@hanshenrik Do note that you have the same blocksize limitation on btrfs (with a couple of extra ways to work around it) so you still need to know what kind of files (large or small or both) you're going to be storing and tune your filesystem accordingly if you want to squeeze the maximum amount of storage out of it. The availability of automatic compression does help quite a bit though if you're storing fluffy data. – Perkins – 2018-08-28T20:23:51.877