Partions, RAID, LVMs, and Encryption

Question

I have an existing production server that is running 4 drives each 750 GB in size in two software RAID levels. RAID 1 for the /boot partition and RAID 5 for the rest of the partitions including the /share volume that is used for serving files to the network. There were no LVMs originally used since it seemed to be too complicated to setup. The box is currently running on Ubuntu 8.04 LTS with the possibility of either moving to Ubuntu 10.04 LTS or a Red Hat derivative of some kind.

The bare metal server is intended to run CIFS, NFS, and possibly KVM.

I have 6 new drives that are each 2 TB in size that I am going to install into this production server and upgrade the OS with a clean install of the OS. I am looking for recommendations on how to choose the software RAID level to use for the new drives and possibly reuse the old drives in a software RAID configuration as well. The follow are some additional considerations that I would like ideas or thoughts on:

• Should the partitions reside on the physical drives or reside within the space that exists after the RAID is created? Should it reside within the LVM that sits on top of the RAID configuration?
• Should I consider the use of RAID 6 in this instance since there will be many files that are important enough such that loosing two drives could be significant?
• I would like to have the new drives running a software RAID level, then have some LVMs that reside on top them, and apply some type of encryption to the data on top of all of this. Is this a good idea or a bad idea? Will this make recovery more difficult later? Is there anything else I should now about this possible setup?
• Is there going to be a significant performance impact when running software RAID, LVM, and encryption?

Thanks in advance for your help and ideas.

Answer 1

I'll answer your questions in order.

o) If you plan to reuse the old drives I'd keep their layout as is (or changed RAID 5 to RAID 6 for the non-boot part), and put a Linux-RAID partition spanning whole disks onto the new 2 TB ones and made RAID 5 or 6 out of them. With such large disks you are running a reasonably high risk of encountering a read error while recovering from a failed drive, so RAID 6 is a very tempting idea.

If you both serve files and host VMs, you might make a RAID 10 out of old disks to hold your system and VMs (RAID 10 is going to give you better performance) and RAID 6 out of new disks for the served files. You might go the other way round if you need more space for VM images than for served files.

On the RAID space I'd put encrypted volumes. It's made easy by modern distros' installers, just remember that your machine will not reboot without human intervention (providing password for the volume). That way all your data is protected from unauthorised access if it's stolen.

The next layer would be LVM because of flexibility it gives, not least because it allows you to fsck a read-only snapshot and find possible silent data corruption without bringing your whole system off line. I think you would do well with two volume groups, one made of old disks, one with only the new ones, this way you know what kind of performance you can expect from which parts of your system. If you don't care or it's not going to be in any way relevant, you might do with just one volume group. I like to have my system and data separated, so I'd made at least three logical volumes into one group (to host swap, / and data) and one in the other (just for data, on the newer, faster disks). Logical volumes are block devices, so you may create filesystems directly on these.

o) RAID 6 should be considered. md will fail recovery if it encounters read error while resyncing array after drive failure, so you may want to be doubly insured. RAID 6 will cost you performance, and it is not a replacement for backups.

o) If you want your data to remain confidential after disks are stolen/disposed then encryption is definitely a good idea. It will cost you performance, because of encryption related computation, but if it's going to be acceptable or not is going to depend on the load you are going to have and CPU power of your hardware. If the server is in a locked, secure room, it may be overkill. If it's in a closet where somebody fast enough could just grab it and flee, it's a very good idea. Any layer you add to the configuration (RAID, LVM, encryption) increases complexity of recovery, so you should plan ahead and have proper tools prepared. I think (not know) that most modern recovery CD distros will just ask you for the password for the encrypted volume, and I know they will take LVM and RAID in their stride.

o) You already know what kind of performance you can expect from software RAID -- you run one :). LVM is not going to be a significant overhead, and encryption is CPU-intensive. You should know if your current/expected load is CPU-intensive and plan to have room to spare. Remember, that any disk access is going to cost you extra cycles now, including any I/O from your VMs. CIFS/NFS servers usually have plenty of cycles to burn, I don't know what kind of load you plan to run on your VMs, so cannot comment there.

Answer 2

I would set it up with the new disks similar to how you have it now, with a couple of exceptions. Partition the 2TB disks into a RAID 1 /boot volume and the rest to RAID 6. If you're mostly planning on using it for CIFS/NFS sharing this should be fine, though if you have KVM guests that are IO intensive or plan on using the disk for other more demanding purposes you may want to reconsider... just depends. There are lots of threads on the benefits and drawbacks of RAID 6 out there, but if multiple redundancy is a priority it may be a good way to go. It's NOT a good reason to not have good backups, though. Remember, RAID != backup!

Place the encrypted volume directly on top of the RAID 6 partition that will store your data, using all of the available space. In the Ubuntu installer, all of this is easy by choosing to manually set up the disks and selecting to use the volumes for RAID, for encryption, LVM, etc, in a step-by-step fashion.

Create your LVM volume group or groups directly on top of the encrypted volume. I usually create 1 group and 2 volumes to start with, 1 for swap and 1 for /, and add more volumes or groups as necessary for your share files, for KVM guest disk volumes, or for whatever else you need specific to the server's purpose.

This approach allows you the redundancy of RAID at the lowest level, encryption on ALL of your non-boot data (including swap), and the flexibility of LVM volumes to manage your data on top of this. There may be some performance hit, but we run this kind of setup (though mostly with RAID 1) a lot and don't have any issues... again, depends on your hardware and your needs.