1

I need to create a private cloud and I'm searching for the best setup. These are my 2 most important requirements 1. Disk and system redundant 2. Price / GB as low as possible

The system is going to be used as backup setup which will receive data 24/7 over SFTP and rsync. High throughput is not that important.

I'm planning to use glusterfs and consumer grade 4TB hard-drives.

I have worked out 3 possible setups

  1. 3 servers with 11 4TB HDD Setup up a replica 3 glusterfs and setup each hard drive as a separate ext4 brick. Total capacity: 44TB HDD / TB ratio of 0.75 (33HDD / 44TB)

  2. 2 servers with 11 4TB HDD The 11 hard-drives are combined in a RAIDZ3 ZFS storage pool. With a replica 2 gluster setup. Total capacity: 32TB (+ zfs compression) HDD / TB ratio of 0.68 (22HDD / 32TB)

  3. 3 servers with 11 4TB consumer hard-drives Setup up a replica 3 glusterfs and setup each hard-drive as a separate zfs storage pool and export each pool as a brick. Total capacity: 32TB (+ zfs compression) HDD / TB ratio of 0.68 (22HDD / 32TB) (Cheapest)

My remarks and concerns: If a hard drive fails which setup will recover the quickest? In my opinion setup 1 and 3 because there only the contents of 1 hard-drive needs to be copied over the network. Instead of setup 2 were the hard-drive needs te be reconstructed by reading the parity of all the other harddrives in the system. Will a zfs pool on 1 harddrive give me extra protection against for example bit rot? With setup 1 and 3 I can loose 2 systems and still be up and running with setup 2 I can only loose 1 system. When I use ZFS I can enable compression which will give me some extra storage.

vdrmrt
  • 101
  • 1
  • 5

1 Answers1

0

Disclaimer: I have no experience with glusterfs.

If you plan to use ZFS please make sure you have a lot of RAM. I've got 5 x 2TB HDDs in a RAIDz1 and it uses about 10Gigs of RAM. So a good rule of thumb is to spend at least 1GB RAM per TB HDD.

A ZFS on a single Harddrive won't give you any protection against "bit rot".

I my opinion you should think about smaller RAIDz2 Pools connected to one big RAIDz2 or RAIDz3. This gives you a lot of protection.

m4rcs
  • 49
  • 3
  • -1: ZFS on a single drive does protect against bit rot. ZFS will let you know that a file is damaged when you try to read or write it. In addition, you can set `copies=x` where x > 1 and ZFS will automatically repair data corruption. – longneck Jun 25 '13 at 16:56
  • But I agree that one pool per disk is a bad idea for this use case. – longneck Jun 25 '13 at 17:00