15

Using a striped mirror (Raid 10) as an example, if two disks are expanded to four, how can the data from the two disks be 're-balanced' (spread out) amongst the two mirrors?

That is, how can the files which were written to one mirror be written to two mirrors to take advantage of the striped disks?

Greg
  • 1,557
  • 5
  • 24
  • 35
  • 9
    @ewwhite Sometimes you don't have that option. – Moshe Katz Jul 06 '17 at 19:45
  • 7
    @ewwhite And sometimes growth cannot be anticipated correctly, because it depends on the outside market. Your advice would only be valid if you also had a working time machine. – user121391 Jul 07 '17 at 07:12

2 Answers2

12

Only newly written (appended) data is split between all currently active vdevs, there is no explicit rebalance operation in ZFS.

Two possible solutions:

  1. Wait until all old data is written again (because of CoW, this can take a very long time, in the worst case double the time it would take to write the disk completely).
  2. Remove all data and write it anew (zfs send/recv is helpful in getting all data off the pool and back without losing anything). This does not have to be done all in one go, and it can be done on the same pool.

To be more precise, I would choose the second solution and transfer each file system separately at times where system load is low (for example at night):

  • Take a snapshot (zfs snapshot -r) of a decently sized file system (and decendant file systems, recursively)
  • Use zfs send | zfs recv with appropriate options to send the snapshot to a newly created temporary file system (can be on the same pool if space permits); this file system should be on the same location in the hierarchy as the old one
  • After the copy is done (may take some time, because the disks have to read and write), zfs destroy the old snapshot and old file system
  • zfs rename the temporary system to the old name
  • Check and change mount points with zfs mount, rearranging the previous situation for your replaced file system
  • Repeat until all file systems are moved
user121391
  • 2,452
  • 12
  • 31
6

possible 3rd solution (as mentioned by SirMaster in this FreeNAS forum post) :

  • add new disk(s) to zpool
  • copy many files to another new /mnt/pool/temp/ directory
  • delete original files: rm -rf original/
  • rename dir back: mv temp/ original/

This works because ZFS will proportionally place writes to whichever vdev has most free space, in this case the brand new drives which were empty. (as of 0.7, zfs will favor faster drives for writes, but lets assume your 2 new drives are same or greater performance than the original drives)

It's probably slower than zfs send | zfs recv, but simpler because you don't have to create/destroy snapshots.

You can run zpool list -v before and after to see each vdev utilization.

Also, found a php script that does the copy/delete/rename procedure on a file-by-file basis. This was linked in an answer from a similar (but zvol) question a few years older. (didn't test this script personally)

jason_uruguru
  • 61
  • 1
  • 2
  • 2
    Thanks, I used this technique to rebalance my home lab after expanding the pool with a new raidz1 vdev. Worked a trick. – Dom Zippilli Aug 09 '20 at 02:53
  • Be aware - this approach may *not* work as you would expect...see Cyberjock's answers down in the thread linked in this post. – quantum Mar 10 '22 at 14:42
  • @quantum That "cyberjock" dude from a decade ago seemed to be making stuff up. He could never give never answer any specific questions about IOPS after rebalancing beyond claiming he's some sort of expert. – Navin Mar 13 '22 at 08:45