ZFS iSCSI and free capacity in VMFS

Question

I use ZFS in FreeNAS 8.0.4 and use iSCSI to provide a LUN to Vmware ESXi 5.0.

I post it at http://forums.freenas.org/showthread.php?10990-Memory-utilization-and-performance-problem/page4, but I believe people here may have more expertise on ZFS.

FreeNAS doesn't have iSCSI unmap. I have create / move / delete virtual machine between ZFS iSCSI. After a while, zpool list report 95% capacity used. However, Vmware shows VMFS only uses 50% capacity.

AFAIK, when creating a virtual machine, zvol allocated disk space. However, when we moved / deleted virtual machine, zvol didn't update / didn't aware to update the free capacity.

[root@data1] ~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data1 2.25T 2.07T 185G 95% ONLINE /mnt

Now, I believe due to the wrong free capacity reported in zpool list, the performance is deeply affected.

How I should do to let ZFS know the real free capacity.

Thanks million.

zpool status -v

  pool: data1
 state: ONLINE
 scrub: scrub stopped after 10h31m with 0 errors on Wed Feb  6 13:32:53 2013
config:

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
        spares
          ada5p2    AVAIL   

errors: No known data errors

zpool list

NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data1  2.25T  2.08T   179G    92%  ONLINE  /mnt

What's the actual problem you're having?? – Chris S Feb 18 '13 at 15:46 — Chris S, Feb 18 '13 at 15:46

score 2 · Answer 1 · edited Apr 13 '17 at 12:14

2

You made your zvol too large and your ZFS implementation doesn't have the ability to unmap release blocks!

The best way to prevent this from happening is to underprovision the LUNs you present to VMware. Since we know ZFS performance degrades when the disk is near full, it's best to limit the maximum size of your zvol.

e.g. if you have 200GB zpool storage, make sure the zvols you present to VMware are less than that - maybe 150GB.

ZFS is unaware of the filesystem being using on top of the zvol block device, so all that's happening on your VMFS volume is that it knows that it has 50% free space. ZFS has no way of knowing that. You will not be able to reclaim the space without reformatting/recreating the volume or expanding the zvol.

A better solution in your case would be using NFS instead of iSCSI for this purpose. If you're not monitoring VM disk utilization and growth carefully enough to prevent this from happening with iSCSI, NFS may make more sense. I use NFS with my ZFS+VMware setups.

Also see: When using thin-provisioning with ZFS, how do you make sure you don't run out of physical disk space?

and

Incorrect Used Space reported by zVol.

edited Apr 13 '17 at 12:14

Community

1

answered Feb 10 '13 at 17:12

ewwhite

194,921
91
434
799

Thanks for your tip of making zvol smaller than zpool. Now, I only want to reclaim the space to increase performance a bit. Then move to another zfs. It's too slow to move now. – Datnus Feb 11 '13 at 03:08
I don't think you have any options. – ewwhite Feb 11 '13 at 03:21
I still have 1 spare disk. If I add it to zpool, will it help? Will the performance be affected to add new HDD to zpool? – Datnus Feb 11 '13 at 03:27
Depends. Show us the output of `zpool status -v`. – ewwhite Feb 11 '13 at 03:34
Bummer. RAIDZ can't be expanded. – ewwhite Feb 11 '13 at 04:39
Sigh, no more luck? – Datnus Feb 11 '13 at 11:16
@Datnus You don't really have an option other than moving your data. – ewwhite Feb 12 '13 at 15:00
I don't really understand. If I use NTFS to give to Windows, would ZFS have this problem? I believe ZFS also doesn't know how much space left in NTFS? – Datnus Feb 18 '13 at 15:14
@datnus Your iSCSI export to Windows needed to be smaller than your actual zpool size. You wrote too much to your Windows iSCSI drive. You won't be able to reclaim the space because ZFS doesn't know (or care) what's inside of your zvol. – ewwhite Feb 18 '13 at 15:37
Let's say. If my zpool is 1TB and my iSCSCI export is 100GB. If NTFS writes and deletes many many times, eventually, zpool will be fooled as FULL? – Datnus Feb 18 '13 at 15:58
No, it will probably reach a ceiling of 100GB. – ewwhite Feb 18 '13 at 16:02
It would be around 100 GB, ASSUMING no snapshots. Throw snapshots in there, and that could quickly go well over 100 GB. Look into 'sdelete', btw. If you have compression on on the dataset (which you should), and no snapshots, sdelete will 'reclaim' most of the space on ZFS that isn't actually used by the filesystem on top through virtue of the zerofilling it does. Of course, this is an active task - you'd need to do it any time you wanted to reclaim space. – Nex7 Mar 22 '13 at 14:45

score 2 · Answer 2 · answered May 10 '13 at 02:33

You have rightly identified the cause - and by the by, even with SCSI UNMAP, it likely would have still occurred, as the support matrix right now for TRIM and UNMAP is abysmal. However, there is a solution nobody has touched on.

First - enable compression (compress=on or compress=lzjb, I'm not a fan of compress=gzip, especially for VM's), none of this will work without that.

If you're dealing with drive(s) offered up into a VM (or as the VM's OS drive, too) and then formatted with a filesystem, you can use utilities such as "zerofree" for Linux or "sdelete" for Windows to write zeroes out to all the free blocks. Due to a quirk in how ZFS handles blocks of 0's on compressed filesystems, this will free up the space on the zvol, getting the zvol much more in line with the actual used capacity.

If you've done the above and it didn't work (unlikely) or there's still a very large discrepancy (also unlikely), even after running this on all drives on all VM's sitting on top of the LUN, you might try this article - note I haven't tested this, at all, so no promises: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

FWIW, at least Solaris also supports `zfs set compression=zle ...` which enables *"zero length encoding"* by just writing zero block pointers for zero-filled blocks. It does not do any other compression, thus the impact on the CPU is minimal. I do not know if the BSD implementation includes support for `zle` though. — the-wabbit, Oct 13 '13 at 07:53

score 0 · Answer 3 · answered Aug 28 '20 at 19:32

0

As you said that

FreeNAS doesn't have iSCSI unmap

maybe you could look into TRIM/UNMAP Zvol over iSCSI in which I asked for help to do exact this: "Reclaim free space on a zfs emulated volume."

answered Aug 28 '20 at 19:32

CodyMouse

75
5

ZFS iSCSI and free capacity in VMFS

3 Answers3