2

I've got an existing Solaris ZFS NAS/SAN device and for a couple of reasons I'm looking to effectively destroy and re-create the zpool. The catch is that I don't have anywhere to put the data elsewhere, so it effectively has to stay on the box while I do this.

There's 12x1TB 7.2k SATA drives which make up the zpool in 6 mirrored pairs. My plan is as follows:

  1. In mirrored pair one, remove one of the drives from the pool completely.
  2. Create a new zpool containing only this drive
  3. Repeat (1) for each of the other mirrored pairs, adding them to the new zpool.
  4. By this point, I'm left with two zpools of the same size, with no redundancy, but containing 6 drives each.
  5. Copy the data over to the new pool and verify (with a scrub, I guess?)
  6. Once the data is copied, destroy the old pool, and add the 6 drives to the new pool as mirrors to the existing 6 drives.
  7. Ensure all the filesystems are correct and have the right attributes

Is there any reason why this wouldn't work? If not, is there a different/better way I could be doing this?

growse
  • 7,830
  • 11
  • 72
  • 114

2 Answers2

2

Why do you need to rebuild the pool? How much data is actually contained in the pool? That will have a greater effect on your migration strategy than anything else. If it is possible to place the data on a single disk (e.g. 1 or 2 Terabytes), I'd do that rather than risk a stripe of unmirrored disks. But again, it depends on how much data needs to move.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • It's about 4TB of data. I need to rebuild to remove some pool attributes such as deduplication, which have been causing a few performance issues. I've been advised dedup in particular can only be purged from the pool by starting afresh. – growse Jun 19 '11 at 18:29
  • It sounds like you need more RAM or L2arc cache. What is the output of `zpool status -D` – ewwhite Jun 19 '11 at 18:33
  • Output in http://pastebin.com/bhpcRh6X - I never even knew about the -D switch giving the DDT size! I should mention - 30GB RAM in the server. Basically, Dedup isn't buying me much, but is costing me a lot, so I want to get rid. – growse Jun 19 '11 at 18:53
  • If this is really all about deduplication, first we'll need to know how you're storing data on the array. Are you serving NFS files? CIFS? iSCSI? Is this a backend for a VMWare environment? – ewwhite Jun 19 '11 at 19:09
  • It's a backend for a small VMware lab (NFS) as well as serving a couple of LUNs for a MSSQL cluster (iSCSI) and being a general fileserver for a small team (CIFS). I only enabled dedup to get the benefits for the CIFS share filesystem, but now have the situation where certain pool-heavy operations (deleting snapshots) has the effect of hanging the entire array. See related http://serverfault.com/questions/274394/why-does-doing-large-deletes-copies-moves-on-my-zfs-nas-block-all-other-ios for my previous question on this. – growse Jun 19 '11 at 19:18
  • I've been through this. The CIFS store is the biggest issue. Turn off dedupe at the pool and filesystem levels. If you're seeing that deletes and snapshot removal is stalling the array, then it's an issue with RAM. Dedupe is handy when the data isn't removed or deleted often. For the flat files on the system, can you rsync them off and back onto the share? Do you have an L2arc device on the array? Also see my question at: http://serverfault.com/questions/234475/zfs-destroying-deduplicated-zvol-or-data-set-stalls-the-server-how-to-recover – ewwhite Jun 19 '11 at 19:39
  • I've turned off dedup everywhere for a while, but still see the same issues. If I can go through the array and over a period of time ensure that all data is moved off and back on (although not all at the same time), will that reduce the DDT to 0 and solve the stall issues? – growse Jun 19 '11 at 19:44
  • Yep. In my case, I had to wait until the data was deleted naturally (it was a backup server). The other thing that helps is an L2ARC device. Get an SSD and assign it to the relevant pool. That provided some relief for me. – ewwhite Jun 19 '11 at 20:43
  • I can probably move the VM data off one VM at a time. The CIFS data I can probably take out of commission for half a day as I move it off (only about 1TB worth) and put it back. – growse Jun 19 '11 at 20:57
0

Your steps seems to amount to "leap frogging" the mirrors which are already in place. That will probably not "undo" the reduplication as those attributes will probably continue forward as the mirror sets rebuild. You may need to consider application-level replication/copying between the deduped vs. non-deduped pools in order to expand-out the actual file instances.

user48838
  • 7,393
  • 2
  • 17
  • 14