17

I have a ZFS server with 8 zpools. Each pool is 12 6TB disks in a 10+2 RAIDZ configuration. So, each pool has a raw storage space of 12*6=72 TB and usable space of 10*6=60 TB. However, I see different results when I query it using zfs list vs. using zpool list as below:

# zfs list
NAME     USED  AVAIL  REFER  MOUNTPOINT
intp1    631K  48.0T   219K  /intp1
intp2    631K  48.0T   219K  /intp2
intp3    631K  48.0T   219K  /intp3
jbodp4   631K  48.0T   219K  /jbodp4

# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
intp1     65T  1.02M  65.0T         -     0%     0%  1.00x  ONLINE  -
intp2     65T  1020K  65.0T         -     0%     0%  1.00x  ONLINE  -
intp3     65T  1.02M  65.0T         -     0%     0%  1.00x  ONLINE  -
jbodp4    65T  1.02M  65.0T         -     0%     0%  1.00x  ONLINE  -

Could someone help me understand why is this discrepancy?

user
  • 4,267
  • 4
  • 32
  • 70
Ketan Maheshwari
  • 323
  • 1
  • 3
  • 8

1 Answers1

14

I'm surprised you have such a large setup. Did you build this array? This is potentially a bad arrangement for performance due to the pool design.

Either way, the zpool man page explains this. zfs list will show your usable space. The zpool list shows parity space as storage space. 

   used                Amount of storage space used within the pool.

   The  space  usage properties report actual physical space available to the storage pool. The physical
   space can be different from the total amount of space that any contained datasets can  actually  use.
   The  amount  of  space used in a raidz configuration depends on the characteristics of the data being
   written. In addition, ZFS reserves some space for internal accounting that the zfs(8)  command  takes
   into  account, but the zpool command does not. For non-full pools of a reasonable size, these effects
   should be invisible. For small pools, or pools that are close to being completely  full,  these  dis-
   crepancies may become more noticeable.
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Thanks! Could you elaborate on the first part. I am still debating this design. What are the cons of this arrangement in your opinion? About the second part, I did consider that zpool must be showing the parity space. However, shouldn't that be 12x6=72TB as opposed to 65TB? PS. I recognize you from your recent HN post on ZFS+NAS :)! Nice work. – Ketan Maheshwari Aug 15 '16 at 00:16
  • 1
    For your accounting needs, use `zfs list`. That's really what matters. For the pool design, it depends on the use case. What do you want to accomplish? What are your performance and capacity requirements? In general, [_for raidz2, do not use less than 6 disks, nor more than 10 disks in each vdev (8 is a typical average)._](http://nex7.blogspot.com/2013/03/readme1st.html) – ewwhite Aug 15 '16 at 00:34
  • 1
    @ewwhite Nice read, but why follow that advice? It's not from Oracle or the zfsonlinux people. What is this 6-10 limit based on? – Ryan Babchishin Aug 15 '16 at 03:30
  • 2
    @RyanBabchishin The author is an engineer at [Nexenta](https://nexenta.com/products/nexentastor), a commercial ZFS storage software vendor. The recommendations are based on experience, performance and thorough understanding of the filesystem. I'm not saying people can't configure things the way they want or that it won't work: It's just not optimal. In the case of the OP, his random I/O performance will suffer under the 12-disk design. – ewwhite Aug 15 '16 at 03:33
  • @wwhite Nexenta, got it... it would be nice to see an official article from them rather than a blog post from their employee. – Ryan Babchishin Aug 15 '16 at 03:34
  • 4
    I think you forgot that 6Tb disk isn't really 6Tb, because there's a conspiracy among disk manufacturers, as they think that one gigabyte isn't 1 073 741 824 bytes, as it should be, but is instead 1 000 000 000 bytes. This takes some significant space from accounting. – drookie Aug 15 '16 at 05:26
  • 2
    @drookie, you are spot on. `lsblk -b` gave me the byte size of each disk which is 5.4TB and not 6TB as I previously thought. – Ketan Maheshwari Aug 15 '16 at 16:37
  • @drookie By that same vein, `Tb` would mean *terabit* and not *terabyte* whereas the OP clearly uses `TB` meaning *terabyte*... – user Aug 17 '16 at 14:18
  • 1
    @drookie the disk manufactures are actually right, 1TB = 1000^4th bytes. But the os tools sometimes shows us TiB (tebibyte) = 1024^4 bytes, and labeling it TB, so, the os tools are wrong imho. see https://en.wikipedia.org/wiki/Tebibyte – Jens Timmerman Aug 24 '16 at 10:37