6

I have a VMware ESXi 4.1 server which I recently added 2x1TB SATA drives to. This machine runs a NexentaStor Community VM which hosts ZFS filesystems. Before adding the new drives, all the ZFS zpools resided inside virtual diska (VMDK files) which resided on the server's Vmware Datastore which is on a hardware RAID10.

The new SATA drives have no hardware redundancy, so my goal was to attach them directly to the NexentaStor VM and create a RAID1 zpool out of them.

I followed these instructions to create two Physical RDM files for the new SATA drives using vmkfstools -z /vmfs/devices/disks/idnumber RDM1.vmdk -a lsilogic

After adding the two RDM disks to the VM and creating a raidz1 zpool on them I started copying data to the zpool. The pool was taken offline and I was informed there were thousands of checksum errors.

I searched the web and found a number of people complaining of the same situation. (Example) I have since given up on using RDMs and have created two datastores and two 930GB VMDK files which I will place in a RAIDz1. However I want to know where I went wrong. A lot of people online said they have this configuration working.

My goals behind using RDMs as opposed to VMDKs were:

  1. Give the VM the ability to monitor S.M.A.R.T. status
  2. Allow ZFS access to the entire disk (since I knew these wouldn't be used for anything else)
  3. Make the drives easy to hotswap should one go bad
  4. Allow me to remove these drives and place them in another ZFS server should I need to

I had planned to use this same setup in a brand new ESXi 5.1 server which I will be setting up later this week. In that case #4 is particularrly important because I wanted to add an existing zpool to a new VM. Why did I get these checksum errors? Is it related to ESXi 4.1? Is there something I did wrong?

Edit: I have created the same setup with VMware ESXi 5.1 and have no issues so far. I'm going to test this extremely thoroughly but so far this appears to be an ESXi 4.1 issue.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
Josh
  • 9,001
  • 27
  • 78
  • 124
  • But the right solution is still to use PCI or controller passthrough. – ewwhite Feb 07 '13 at 23:55
  • I'd still like to understand why that is. Can you point me to a document explaining what the proper usage of RDMs is? (Not to mention that I cannot use PCI passthrough for my SATA controller based on physical hardware configuration) – Josh Feb 08 '13 at 01:33
  • For example I have seen [Chopper3](http://serverfault.com/users/1435/chopper3) recommend RDMs in other serverfault posts. I'm looking for VMware documentation on when they should/should not be used. I'm trying to learn, looking for more than "don't do it". – Josh Feb 08 '13 at 01:42
  • RDM's are used in situations where they are necessary (large volumes, database clusters, cluster filesystems, SAN-backed solutions). Your situation would benefit most from a ***dedicated SAS controller***. I've linked you to two examples in my answer. Also see [here](http://blogs.vmware.com/vsphere/2013/01/vsphere-5-1-vmdk-versus-rdm.html). But you don't have to take my word for it... – ewwhite Feb 08 '13 at 01:58
  • Thanks @ewwhite I will read that tomorrow. Also, please don't misunderstand, I'm not *doubting you*, I'm just trying to broaden my knowledge, that's all. No offense intended! – Josh Feb 08 '13 at 02:29

2 Answers2

5

Is it strongly recommended to avoid using RDM's for this purpose.

Pass your SATA controller through to the virtualized NexentaStor instance using "PCI Passthrough". This requires a reasonably-current processor to do so.

See: Hosting a ZFS server as a virtual guest

That said, there's no major issue running ZFS on VMDK's if you have an underlying hardware RAID. Can you provide more information about the hardware solution?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • What if I cannot pass my SATA controller through? For example, in the case of the new machine I will be setting up, I will need to install VMware ESXi to a SATA drive. – Josh Feb 06 '13 at 15:40
  • Well, what you're doing may not be possible. You could install ESXi to a USB drive, but you'll still need persistent storage to house the Nexenta VM. So in this case, you need another controller if you want full-disk access... OR a hardware RAID to house the full setup and run ZFS on a VMDK. – ewwhite Feb 06 '13 at 15:51
  • OK... I'll see what options I have. Can you explain why RDMs don't work like this or how others made it work? Specifically, I'm trying to understand how RDMs work and what they can be used for / why they failed in this configuration. – Josh Feb 06 '13 at 17:02
  • People don't use VMware RDM's for this purpose. – ewwhite Feb 06 '13 at 19:14
0

As far as I can tell, this was caused by a bug in the RDM implementation in VMware ESXi 4.1. Using VMware ESXi 5.1 I see no issues with RDMs at all. I'd still like to understand more about why the RDMs didn't work as I expected or what they are used for, but using RDMs on 5.1 has been very stable and allows me to achieve my goals.

Josh
  • 9,001
  • 27
  • 78
  • 124