3

I have an HP DL360 G7 server I plan to install KVM and ZFS on. The purpose is going to be a lab in a box. I have the 8 drive bays in the front loaded with 4 drives and an SSD (for the ZFS ZIL cache). My goal was to keep the disk array away from the actual OS disk. What I am wonder is weather or not it is safe to boot a Linux server installation from a USB drive for "production" use. The server has an embedded USB / SD card reader on the motherboard for VMware and other embedded solutions. This raises a question to me because once VMware is loaded it stay in memory. On the other hand a Linux install does not (atleast not 100%). I am concerned if I load the OS on a USB drive (or SD card) I will burn the SD card out.

Can anyone please give me some insight on this? I am wondering what my options are. The way I see it my options currently are make Linux boot from the ZFS array or use a USB drive. The first option would be okay if I could make grub play nice with ZFS root booting. Which to my knowledge is broken or at least needs some nasty hack to pull it off.

Like I said any help you could provide would be great.

ianc1215
  • 1,965
  • 7
  • 34
  • 55

3 Answers3

4

You can, but you probably shouldn't.

The DL360 G7 doesn't present disks in a JBOD fashion. If you're using the onboard HP Smart Array controller, this won't work the way you expect.

Depending on the OS you use, there are swap and other I/O activity considerations.

Why go through all of this? If you want ZFS, just use it on the server in a baremetal OS installation. ZFS root is not great... however, you have an HP controller that's capable of multiple logical drives and better reporting/monitoring than you'd get otherwise. The ZIL SSD is probably not even necessary.

See: ZFS best practices with hardware RAID

Can you give a better idea of why you wanted to keep the OS and data drives separate?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I actually swapped out the controller for an lsi 9211-8i. – ianc1215 Jun 23 '15 at 01:42
  • My take... just use hardware RAID with ZFS atop. Otherwise, you have to deal with wasted disk capacity, finding a place to house the OS, needing a ZIL device and less monitoring. If not, just install Linux using the LSI controller and ZFS software RAID. I would not use USB boot unless you are willing to deal with a failure of the boot device. You could also partition the SSD for boot/ZIL duties. – ewwhite Jun 23 '15 at 01:46
  • well I originally wanted to keep it separated for format reasons. if I install Linux on an ext4 partition then I can't dedicate the drive to zfs. – ianc1215 Jun 23 '15 at 01:46
  • yeah I think I will just install everything on the raid. do you know if modern grub2 packages support zfs root? – ianc1215 Jun 23 '15 at 01:48
  • I don't deal with or recommend ZFS root. But the [ZFS mailing list](http://list.zfsonlinux.org/mailman/listinfo/zfs-discuss) may be able to help. – ewwhite Jun 23 '15 at 01:55
  • I don't recommend putting ZFS on top of hardware RAID, and I'm not the only one. You lose almost all the data integrity benefits of ZFS if you put it on top of hardware RAID. Far better to have a separate OS disk on USB. – noitsbecky Jun 23 '15 at 09:23
  • @qasdfdsaq ZFS isn't just about data integrity. It's an excellent volume manager. And in cases where boot, hardware support and other considerations are factors, you can't make a blanket statement like that. – ewwhite Jun 23 '15 at 09:27
  • @ewwhite: I can exactly make a blanket statement like that because it is 100% correct. – noitsbecky Jun 23 '15 at 09:32
3

Is it safe to boot Linux from a USB drive?

Yes, why not? I do it regularly. But having said that, it's not entirely trivial to do it well. If you just straight install Linux to a USB disk it's usually really slow and doesn't really work that great.

However, if you use a tool like unetbootin it'll create a more optimal installation with which you can also install some configuration.

FreeNAS actually is designed to run this way, so that you have all of your HDD's and SSD's available for storage and the OS is booted off flash onto a RAM disk. Although FreeNAS is not Linux based, I mention it here to tell you this is not an unusual solution and is used in production.

When/if you do get it to work, make sure you back up your USB stick every time you make changes to it's stored configuration.

hookenz
  • 14,132
  • 22
  • 86
  • 142
  • (This isn't FreeNAS) – ewwhite Jun 23 '15 at 09:42
  • @ewwhite - Obviously! at least to you and I. It's also FreeBSD based. I mentioned it here, because flash booting is a practical thing to do in a production environment and widely done. – hookenz Jun 23 '15 at 19:58
  • I don't think it's that practical anymore. Again, I'm coming from the VMware world, but I've seen tremendous failure counts at scale with ESXi [(_which is optimized for USB/SD_)](http://serverfault.com/questions/549253/what-happens-when-the-usb-key-or-sd-card-ive-installed-vmware-esxi-on-fails/550282#550282), so the novelty of flash boot isn't worth the time for me. – ewwhite Jun 23 '15 at 20:05
  • Sounds like despite the claim, ESXi is just not optimized for USB/SD. It should boot to RAM disk and only store config data on the USB/SD. If it's not doing that, then it's going to wear out the USB stick very quickly. – hookenz Jun 23 '15 at 20:08
  • He mentions ESXi, but it sounds like that's not the scenario he is planning to use it in anyway. – hookenz Jun 24 '15 at 02:40
3

Define "safe". As in:

  • Does the system boot from USB? Yes
  • Data integrity? No
  • Fault tolerance? No

I would not trust a file server to be booted from a single USB pen, unless it's a server just for testing/lab etc. We use USB drives (expensive ones) in our vSphere Cluster, and I've encountered failures many times.

You've indicated that you have replaced the built-in smartarray controller with a LSI HBA card. If I were you I'd dedicate two drives (2x200GB MLC SATA SSD or similar) for the operating system + swapfile, and create some sort of mirror on them. I have no idea if Linux can boot from ZFS in a stable manner, so you should consider using the operating systems native software raid with a common, trusted filesystem on top (ext3, ext4, xfs).

pauska
  • 19,532
  • 4
  • 55
  • 75
  • Yeah, the OP sounds like they do not have enough resources to do this with the right number of disks, but this is the answer. Definitely use a pair of OS drives. – ewwhite Jun 23 '15 at 13:09