8

I need to expand disk capacity of the server. The pool was started with 1Tb disk, then it was expanded with 2Tb disk. There is more than 1Tb free space, i.e. all data will easily fit 2Tb part, but it is currently allocated on 1Tb disks.

In reality, these disks are hardware (PERC) RAID1 arrays over pair of 1Tb and pair of 2Tb resp.

I want to replace these 1Tb disks with 3Tb. The "one by one" replacement isn't really an option. In principle, this physical RAID can replace disks one by one, then grow the array to fill disks. However, I want to avoid this path, because it leaves some quite long time periods when disk redundancy is lost.

I want to literally move all data off from 1Tb, then remove it, and replace with 3Tb. All to be done on the fly, with running system, with zero downtime.

With LVM the operation has to be very straightforward and easy to understand:

  1. pvmove all allocated data from 1Tb physical disk (VD in RAID terms)
  2. vgreduce to remove that pv from vg and pvremove to remove pv metadata
  3. remove a 1Tb array using megacli (PERC is rebranded LSI/Avago MegaRAID SAS)
  4. physically replace disks
  5. assemble another array again using megacli
  6. make a new pv and add it into vg

This is the routine procedure I used to do. Each step is very well understood, on each step I have full control of what is going on, I always know how to proceed if something goes wrong and so on.

How to do the same procedure safely and knowingly with ZFS?

If that matters:

  • the server is Dell PowerEdge R730
  • the OS is Proxmox VE 6.0, which is based on Debian 10.1. It was installed from PVE ISO image, i.e. not converted from Debian installation.
  • the system doesn't depend on this this pool, because it runs from set of SSDs assembled into another pool
  • the pool hosts some VM virtual disks which don't require high performance. However, that data is valuable, I can't tolerate the if it's lost. Therefore the procedure should be clear and understandable
  • system is constantly used by users, but they will tolerate the performance loss during data migration
Nikita Kipriyanov
  • 8,033
  • 1
  • 21
  • 39
  • It looks like you could do this with `zpool remove poolName 1TBmirrorName` - https://docs.oracle.com/cd/E37838_01/html/E61017/remove-devices.html – JimNim Oct 15 '19 at 15:20
  • 1
    It doesn't matter much what model of the server it is happening on, but ZFS pool layout (down to each HDD) does. And why does your ZFS pool use HW RAID1 arrays? – poige Oct 15 '19 at 16:47
  • @poige because I'm new to ZFS and want it to be introduced in small portions. Nobody is able to catch up a complex system in its entirety in one shot, and I am unable in particular. Use a feature and only it and nothing else to interfere with, internalize it — then add another feature, internalize — here comes the next. This is the only structured systematic way to really learn something and be intimate with it. Too bad ZFS is ugly monolithic and its layers can't really be separated, so the only way to learn it systematically is to not use some features and hope they will not interfere much. – Nikita Kipriyanov Oct 15 '19 at 17:05

1 Answers1

5

how to move data off from vdev?

Can't be done except for mirrored pools.

poige
  • 9,171
  • 2
  • 24
  • 50
  • 2
    This article sums all by bad feeling about this hype around ZFS. – Nikita Kipriyanov Oct 15 '19 at 17:08
  • 3
    Well, ZFS has its strengths too! As to the very issue you're having — there're lots of possible ways to solve that gracefully, at least due Proxmox' built-in means for VEs' storage transparent (live) migrations. – poige Oct 15 '19 at 18:14
  • From the question you could have been deduced I know what PVE can do. At least that I can install Debian and convert it to Proxmox ;) Indeed, I work with PVE clusters from at least version 2.0, so don't worry, I know how to solve this by other means. The question wasn't about great Proxmox at all, but about ZFS crippled volume management. I am very disappointed, I wanted to give ZFS a try, but no, it is still not ready for production. It is only ready for homegrown experiments. It definitely needs its strengths for that. – Nikita Kipriyanov Oct 15 '19 at 18:48