ZFS, raidz1: Why is the volume size (42T) so much smaller than the pool size (50T)?

Question

I have a pool of 28 2TB-disks (56T) in 4 arrays of 7 disks. Since it's raidz1 (~RAID5), I'd expect 1 disk to be used for parity in each array, so the resulting volume should be 2TB*4*(7-1)=48TB, right?

Now, what I see on my system:

$ zpool list volume
NAME     SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
volume  50.5T   308K  50.5T   0%  1.00x  ONLINE  -

$ zfs list volume
NAME     USED  AVAIL  REFER  MOUNTPOINT
volume  2.00T  40.3T  75.8K  /volume

$ df -h /volume
Filesystem             Size   Used  Available Capacity  Mounted on
volume                  42T    75K        40T     1%    /volume

So, there are only 42T instead of 48T. Where are the missing 6TB? And where does the number 50.5T come from?

score 6 · Accepted Answer · answered Jan 20 '12 at 18:00

6

A 2TB disk is not 2 TiBi in size - it's only 2*10^12 / 2^30 ~ 1862 GiBi.

4 arrays of 6 effective disks each would be 24 * 1862 = 44703 GiBI, or 43.6 TiBi of real, usable storage.

I reckon it has some additional overhead you're not taking into account - IIRC RAIDZ also does snapshots and scrubbing, which take up additional space.

answered Jan 20 '12 at 18:00

adaptr

16,479
21
33

+1 Rounding, snapshots, reservations, other file systems, etc can all play into the slightly "off" number. Those are functions of ZFS, RAIDZ is a part of zpools, which are part of ZFS. – Chris S Jan 20 '12 at 18:03
1

Note: The binary prefix for "tera-" is "Tebi", not "Tibi". And "Gibi" should not have a capitalized "B". – Chris S Jan 20 '12 at 18:04
That sounds perfectly reasonable! Thank you very much for this explanation, guys! – Pavel Jan 20 '12 at 18:06

Shane Madden · Answer 2 · 2012-01-20T18:18:04.227

5

To clarify the discrepancy in output between the commands:

The zpool command counts the disks that are being used for redundancy as space, while the zfs command does not; thus, the 50.5 TB number is your raw disk size, while the 42T is after taking out the 4 disks for redundancy.

edited Jan 20 '12 at 18:18

answered Jan 20 '12 at 18:02

Shane Madden

112,982
12
174
248

...except the after-RAID figure is 43.6, not 42. – adaptr Jan 20 '12 at 18:10
@adaptr Indeed; as you mentioned, something is fuzzy with its math (though no snapshots are listed so it's not that, and scrubbing is not a space spend - but there's definitely some extra usage somewhere). I saw your answer come up before mine, and only added mine to help clarify the difference in output between the `zpool` and `zfs` commands. – Shane Madden Jan 20 '12 at 18:17

score 1 · Answer 3 · answered Jan 20 '12 at 18:00

1

Hard drive manufactures measure disk size in Base 10. Computers measure bytes in Base 2.
It should say 42 TiB to clarify the SI use of the tera- prefix.

answered Jan 20 '12 at 18:00

Chris S

77,337
11
120
212

ZFS, raidz1: Why is the volume size (42T) so much smaller than the pool size (50T)?

3 Answers3