17

I'm a bit confused about the recent developments in PCIe-based storage, particularly as it relates to the NVMe specification and its hardware compatibility.

While I've worked extensively with SSDs in disk form-factor and some higher-end PCIe devices like Fusion-io, I'm in a position where I don't understand the basics of NVMe and am seeking clarification on what type of server hardware is supported.

For instance, ad copy like this from Supermicro is confusing.

...high performance CPU PCI-E Gen3 direct connnect to NVMe devices.

I'm dealing with a Linux-based software-defined-storage solution and wanted to use spare Fusion-io devices, which use a proprietary driver (presenting /dev/fioX device names to the OS).

When I asked for help from the vendor, the response was:

The "fioX" device naming is made obsolete by the new NVMe device interface. It means us purchasing obsolete adapters to add support that nobody else has asked for.

This seems a bit harsh. I didn't think Fusion-io adapters were obsolete.

The scarce information I find online seems to hint that NVMe is only supported on the absolutely newest generations of server hardware (Intel E5-2600v3 CPUs and PCI 3.0 chipsets?). But I can't verify this.

Is this true?

What's the adoption rate? Is this something that engineers are accounting for in their design decisions, or are we talking about a "standard" that's not fully formed?

If NVMe is something that only applies to the newest systems in the market, is it reasonable to suggest (to the vendor) that my install base of older systems can't be NVMe-compatible, so it's worth adding the support I requested?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799

8 Answers8

4

NVMe is PCIe based, and using different drivers designed for that. You can essentially take an M2 formfactor NVM, pop it into the appropriate adaptor, and run it on any linux, windows or BSD system with appropriate drivers.

Essentially all NVMe does is standardises PCIe based SSDs to a single set of drivers, designed to take full advantage of them.

Chances are if you could boot from a non standard pci ssd, you can boot from this, with appropriate drivers in the OS. If you're using this for caching and other purposes, there's no reason NVMe shouldn't work.

Your older systems probably are compatible, assuming you can get a new enough OS with a modern enough kernel on them.

Journeyman Geek
  • 6,969
  • 3
  • 31
  • 49
3

I needed to test this for myself...

I purchased four Intel 750 PCIe NVMe SSDs to install in HP ProLiant DL380p Gen8 servers. The servers are not the current generation Intel 2600v3 series CPUs, but rather the 2600v2 CPUs.

The takeaway:

NVMe is an interface specification. Under Linux, the devices are enumerated as /dev/nvmeXnY, e.g. /dev/nvme0n1 and /dev/nvme1n1.

The form-factor of the devices I used was PCIe 3.0 x4. The Gen8 ProLiant servers have two PCIe 3.0 slots on the default riser cage. These NVMe PCIe cards will work in slower PCIe slots (or PCIe 2.0), but will be limited by the bus at that point.

So for my use case, NVMe is somewhat OS-driven, but is definitely compatible with my slightly older server hardware.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • ewwhite-- I'm interested in moving to these NVMe cards on our HP gen8 servers as well. Did you have to supply external SATA or other power cables to make things work? We have a mix of OCZ and Mushkin PCIe SSDs but all use SATA3 or SCSI controllers to interface. I'm just beginning to hear more of these NVMe drives becoming more common. –  Aug 03 '15 at 01:17
  • I used NVMe PCIe cards. They slide right in. Only 2 of the 3 slots in a Gen8 PCIe riser are PCIe 3.0, so be careful of that. – ewwhite Aug 03 '15 at 01:19
2

My two cents...

NVMe got the various SSD mfg to focus on and adopt a base standard... Basically you can get Nand Flash performance from an SSD connected to NVMe servers for NET less. Also their is more NVMe over fabric features (that I am not that familiar with yet)

See https://www.brighttalk.com/webcast/663/132761

Content "The Performance Impact of NVMe and NVMe over Fabrics" •An overview of the NVMe over Fabrics initiative •NVMe support for multiple fabrics including Ethernet with RDMA (iWARP) •How NVM Express end-to-end eliminates SCSI translation latency •Achieving performance benefits comparable to hundreds of SSDs – local and remote

William Moore NetDirect CA
  • 21
  • 3
2

this article in PCWorld may be of interest.

Ry Jones
  • 404
  • 8
  • 18
2

I wrote an article at Thinkmate that tries to give a nice overview of NVMe and works as a nice little guide on choosing the right drive and system, something that I've found to be missing online. We mostly sell Supermicro servers, and I agree - Things can get a bit confusing... That is why I wrote the article!

As for adoption, I can't speak for the industry as a whole but at Thinkmate interest in NVMe has been significant, and the feedback we get from our customers is that they're very pleased with their purchase, mostly due to the performance benefits.

I see a lot of talk about price, but I think the most important thing to consider is value, not price. Your choice of flash memory should depend heavily on the value that each solution will bring to your application over the life of the drive and system.

Eli Domingues
  • 21
  • 1
1

I added an intel 750 400GB nvme card to a DL 380p gen 8 with no trouble at all. The bios recognised it as a mass storage device.

I'm using FreeBSD 9.3 and it recognised the nvme drive immediately, works perfectly well and is very fast. I can't comment on booting from the drive as I'm using this as a second drive.

Phil
  • 11
  • 1
  • Be aware that only PCIe slots 1,2,4,5 and 6 are capable of PCIe 3.0 speeds. Slot #3 is a bad place for an NVMe card. – ewwhite Dec 10 '15 at 19:56
1

There's one detail in the following article that I think is significant, and takes it beyond the 'just pcie'.

It is the 32k simultaneous queued requests, which could transform some applications. If I've read it right, that isn't available via the 'fit it in a slot that takes it' solution. It needs a proper NVMe board.

Just for info. I'm learning too!

http://www.pcworld.com/article/2899351/everything-you-need-to-know-about-nvme.html

Andrew Mather
  • 11
  • 1
0

The key question is are you trying to boot off the device?

If you want to boot off the device, then either the device needs to provide an "option rom" which can be used to boot it or the BIOS/UEFI on the firmware on the motherboard needs to understand how to talk to the device. NVME SSDs generally don't come with an option rom, so you can only boot them on systems that are new enough for the firmware to understand NVME.

If the answer is no, then a NVME SSD is just another PCIe device. Nothing special, as long as you can physically connect it (may require an adapter, as most NVME drives come in the M.2 form factor) and your OS has a driver you should be able to use it.

Peter Green
  • 4,056
  • 10
  • 29