I'm going to build a freenas server, would like to make sure what I can do with such magical and advanced zfs.
If I have 5 * 3TB disks in RAIDZ (12TB storage in total), now I am trying to add another 2 * 3TB disks to this existing array.
Q:
As ewwhite said, pool shrinking isn't currently possible with ZFS. If you need to do it, you'll have to backup to another storage medium (another pool, tape, SAN, etc) create a new pool and restore. As for expanding there are a number of options of how to grow your ZFS 5x3TB raidz pool:
Mixing multiple sizes or types of VDEVs (raidz + mirror) is non-optimal performance wise and offers the effective redundancy of the least redundant VDEV (raidz). At home or in a pinch you probably don't care, but it should be avoided if possible.
Realistically though, you shouldn't upgrade this pool. You shouldn't rely on RAID-Z and should switch to mirrors or RAID-Z2. It will cost you an extra disk (RAID-Z2) or two (mirroring) over RAID-Z to reach the same usable capacity, but there's a reason enterprises don't use RAID5 anymore. If a single disk fails, even with a hot spare ready to go it takes forever to recreate the missing disk from parity. Likely 24-48hrs or longer. If a second disk fails during this window (which is not unlikely because you're working the remaining disks as hard as you ever will, reading every byte off every disk as fast as you can) you will loose everything. With RAID-Z2 (think RAID6) two disks can fail without risking data loss. Consider the simplicity of mirrored VDEVs, but if cost/GB is your primary concern (at the expense of performance and expandability) double parity RAID-Z2 really is much safer.
That said, if you can find the temporary space and can afford the downtime, here are some more optimal ways to utilize your seven 3TB disks:
If you'd chosen mirroring over raidz in the beginning, the initial five disks would've yielded a 4x3TB mirror + hotspare (6TB usable, 2 vdevs) and you could've just added pairs of disks as you needed them. Also, in case you're curious write performance is directly correlated with the number of VDEVs in the pool.
Really, it's all about how valuable your data is. If you have everything backed up elsewhere, you don't care about the 5% chance of a second disk failing during rebuild. But if you, like most people considering ZFS for a home server, have come to the conclusion that you won't (or can't afford to) perform regular backups and likely won't notice/replace a failed disk immediately, you should really consider the incremental cost of a hotspare and mirroring/RAID-Z2 instead of simple parity (RAID-Z) it just increases the odds you'll still have your data a few years down the road. ZFS goes to great lengths to use software to keep from loosing your data (checksums, parity/redundant copies, scrubbing, etc) without the need for an expensive and proprietary controller card. The least you can do is give ZFS the means means to protect against failed/failing disks: more than just the minimum number of disks required.
You cannot expand a RAIDZ volume in ZFS by adding disks, nor can you shrink a RAIDZ volume by removing disks.
You can, however, stripe across multiple RAIDZ vdevs. You can also swap individual disks for larger disks and grow a pool that way.
Also see: Where can I find introductory documentation for ZFS?
