NVM Express

NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCIS) is an open logical-device interface specification for accessing non-volatile storage media attached via PCI Express (PCIe) bus. The acronym NVM stands for non-volatile memory, which is often NAND flash memory that comes in several physical form factors, including solid-state drives (SSDs), PCI Express (PCIe) add-in cards, M.2 cards, and other forms. NVM Express, as a logical-device interface, has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices.[1]

Non-Volatile Memory Host Controller Interface Specification
Year started2011 (2011)
AbbreviationNVMe
Websitenvmexpress.org

By its design, NVM Express allows host hardware and software to fully exploit the levels of parallelism possible in modern SSDs. As a result, NVM Express reduces I/O overhead and brings various performance improvements relative to previous logical-device interfaces, including multiple long command queues, and reduced latency. (The previous interface protocols were developed for use with far slower hard disk drives (HDD) where a very lengthy delay (relative to CPU operations) exists between a request and data transfer, where data speeds are much slower than RAM speeds, and where disk rotation and seek time give rise to further optimization requirements.)

NVM Express devices are chiefly available in the form of standard-sized PCI Express expansion cards[2] and as 2.5-inch form-factor devices that provide a four-lane PCI Express interface through the U.2 connector (formerly known as SFF-8639).[3][4] Storage devices using SATA Express and the M.2 specification which support NVM Express as the logical-device interface are a popular use-case for NVMe and have become the dominant form of solid-state storage for servers, desktops and laptops alike.[5][6]

Specifications

Specifications for NVMe released to date include:[7]

  • 1.0e (January 2013)
  • 1.1b (July 2014)
  • 1.2 (November 2014)
    • 1.2a (October 2015)
    • 1.2b (June 2016)
    • 1.2.1 (June 2016)
  • 1.3 (May 2017)
    • 1.3a (October 2017)
    • 1.3b (May 2018)
    • 1.3c (May 2018)
    • 1.3d (March 2019)
  • 1.4 (June 2019)
    • 1.4a (March 2020)

Background
Intel SSD 750 series, an SSD that uses NVM Express, in form of a PCI Express 3.0 ×4 expansion card (front and rear views)

Historically, most SSDs used buses such as SATA, SAS or Fibre Channel for interfacing with the rest of a computer system. Since SSDs became available in mass markets, SATA has become the most typical way for connecting SSDs in personal computers; however, SATA was designed primarily for interfacing with mechanical hard disk drives (HDDs), and it became increasingly inadequate for SSDs, which improved in speed over time.[8] For example, within about 5 years of mass market mainstream adoption (2005–2010) many SSDs were already held back by the comparatively slow data rates available for hard drives—unlike hard disk drives, some SSDs are limited by the maximum throughput of SATA.

High-end SSDs had been made using the PCI Express bus before NVMe, but using non-standard specification interfaces. By standardizing the interface of SSDs, operating systems only need one driver to work with all SSDs adhering to the specification. It also means that each SSD manufacturer does not have to use additional resources to design specific interface drivers. This is similar to how USB mass storage devices are built to follow the USB mass-storage device class specification and work with all computers, with no per-device drivers needed.[9]

NVM Express devices are also used as the building block of the burst buffer storage in many leading supercomputers, such as Fugaku Supercomputer, Summit Supercomputer and Sierra Supercomputer, etc.[10][11]

History

The first details of a new standard for accessing non-volatile memory emerged at the Intel Developer Forum 2007, when NVMHCI was shown as the host-side protocol of a proposed architectural design that had Open NAND Flash Interface Working Group (ONFI) on the memory (flash) chips side.[12] A NVMHCI working group led by Intel was formed that year. The NVMHCI 1.0 specification was completed in April 2008 and released on Intel's web site.[13][14][15]

Technical work on NVMe began in the second half of 2009.[16] The NVMe specifications were developed by the NVM Express Workgroup, which consists of more than 90 companies; Amber Huffman of Intel was the working group's chair. Version 1.0 of the specification was released on 1 March 2011,[17] while version 1.1 of the specification was released on 11 October 2012.[18] Major features added in version 1.1 are multi-path I/O (with namespace sharing) and arbitrary-length scatter-gather I/O. It is expected that future revisions will significantly enhance namespace management.[16] Because of its feature focus, NVMe 1.1 was initially called "Enterprise NVMHCI".[19] An update for the base NVMe specification, called version 1.0e, was released in January 2013.[20] In June 2011, a Promoter Group led by seven companies was formed.

The first commercially available NVMe chipsets were released by Integrated Device Technology (89HF16P04AG3 and 89HF32P08AG3) in August 2012.[21][22] The first NVMe drive, Samsung's XS1715 enterprise drive, was announced in July 2013; according to Samsung, this drive supported 3 GB/s read speeds, six times faster than their previous enterprise offerings.[23] The LSI SandForce SF3700 controller family, released in November 2013, also supports NVMe.[24][25] A Kingston HyperX "prosumer" product using this controller was showcased at the Consumer Electronics Show 2014 and promised similar performance.[26][27] In June 2014, Intel announced their first NVM Express products, the Intel SSD data center family that interfaces with the host through PCI Express bus, which includes the DC P3700 series, the DC P3600 series, and the DC P3500 series.[28] As of November 2014, NVMe drives are commercially available.

In March 2014, the group incorporated to become NVM Express, Inc., which as of November 2014 consists of more than 65 companies from across the industry. NVM Express specifications are owned and maintained by NVM Express, Inc., which also promotes industry awareness of NVM Express as an industry-wide standard. NVM Express, Inc. is directed by a thirteen-member board of directors selected from the Promoter Group, which includes Cisco, Dell, EMC, HGST, Intel, Micron, Microsoft, NetApp, Oracle, PMC, Samsung, SanDisk and Seagate.[29]

In September 2016, the CompactFlash Association announced that it would be releasing a new memory card specification, CFexpress, which uses NVMe.

Form factors

There are many form factors of NVMe solid-state drive, such as AIC, U.2, U.3, M.2 etc.

AIC (add-in card)

Almost all early NVMe solid-state drives are HHHL (half height, half length) or FHHL (full height, half length) AIC, with a PCIe 2.0 or 3.0 interface. A HHHL AIC NVMe solid-state drive is easy to adapt to insert in to the PCIe socket of a server.

U.2 (SFF-8639)

U.2, formerly known as SFF-8639, is a computer interface for connecting solid-state drives to a computer. It uses up to four PCI Express lanes. Available servers can combine up to 4 U.2 NVMe solid-state drives.[30]

U.3 (SFF-8639)

U.3 is built on the U.2 spec and uses the same SFF-8639 connector. It is a 'tri-mode' standard, combining SAS, SATA and NVMe support into a single controller. U.3 can also support hot-swap between the different drives where firmware support is available. U.3 drives are still backward compatible with U.2, but U.2 drives are not compatible with U.3 hosts.

M.2

M.2, formerly known as the Next Generation Form Factor (NGFF), uses a M.2 NVMe solid-state drive computer bus. Interfaces provided through the M.2 connector are PCI Express 3.0 or PCI Express 4.0 (up to four lanes).

NVMe-oF

NVM Express over Fabrics (NVMe-oF) is the concept of using a transport protocol over a network to connect remote devices, contrary to NVMe where devices are connected directly to PCIe bus (or over a PCIe switch to a PCIe bus) In September 2014, a standard for using NVMe over Fibre Channel (FC) was proposed[31] and this combination is often referred to as FC-NVMe (or sometimes NVMe/FC). NVMe-oF is a communication protocol that allows one computer to access block-level storage devices attached to another computer via remote direct memory access over a number of transport protocols:

The standard for NVMe over Fabrics was published by NVM Express, Inc. in 2016.[32][33]

The following drivers implement the NVMe-oF protocol:

  • The Linux NVMe-oF initiator and target drivers[34]
  • The Storage Performance Development Kit (SPDK) NVMe-oF initiator and target drivers[35]
  • An NVMe-oF initiator driver for Microsoft Windows[36]

Starting with Linux Kernel 5.0 native support has been added for NVMe/TCP.

Comparison with AHCI

The Advanced Host Controller Interface (AHCI) has the benefit of wide software compatibility, but has the downside of not delivering optimal performance when used with SSDs connected via the PCI Express bus. As a logical-device interface, AHCI was developed when the purpose of a host bus adapter (HBA) in a system was to connect the CPU/memory subsystem with a much slower storage subsystem based on rotating magnetic media. As a result, AHCI introduces certain inefficiencies when used with SSD devices, which behave much more like RAM than like spinning media.[5]

The NVMe device interface has been designed from the ground up, capitalizing on the low latency and parallelism of PCI Express SSDs, and complementing the parallelism of contemporary CPUs, platforms and applications. At a high level, the basic advantages of NVMe over AHCI relate to its ability to exploit parallelism in host hardware and software, manifested by the differences in command queue depths, efficiency of interrupt processing, the number of uncacheable register accesses, etc., resulting in various performance improvements.[5][37]:17–18

The table below summarizes high-level differences between the NVMe and AHCI logical-device interfaces.

High-level comparison of AHCI and NVMe[5]
  AHCI NVMe
Maximum queue depth One command queue;
32 commands per queue		65535 queues;[38]
65536 commands per queue
Uncacheable register accesses
(2000 cycles each)		 Six per non-queued command;
nine per queued command		Two per command
MSI-X
and interrupt steering		 A single interrupt;
no steering		2048 MSI-X interrupts
Parallelism
and multiple threads		 Requires synchronization lock
to issue a command		No locking
Efficiency
for 4 KB commands		 Command parameters require
two serialized host DRAM fetches		Gets command parameters
in one 64-byte fetch

Operating system support
The position of NVMe data paths and multiple internal queues within various layers of the Linux kernel's storage stack.[39]
Chrome OS
On February 24, 2015, support for booting from NVM Express devices was added to Chrome OS.[40][41]
DragonFly BSD
The first release of DragonFly BSD with NVMe support is version 4.6.[42]
FreeBSD
Intel sponsored a NVM Express driver for FreeBSD's head and stable/9 branches.[43][44] The nvd(4) and nvme(4) drivers are included in the GENERIC kernel configuration by default since FreeBSD version 10.2 in 2015.[45]
Genode
Support for consumer-grade NVMe was added to the Genode framework as part of the 18.05[46] release.
Haiku
Haiku Gained support for NVMe on April 18, 2019.[47][48]
illumos
illumos received support for NVMe on October 15, 2014.[49]
iOS
With the release of the iPhone 6S and 6S Plus, Apple introduced the first mobile deployment of NVMe over PCIe in smartphones.[50] Apple followed these releases with the release of the first-generation iPad Pro and first-generation iPhone SE that also use NVMe over PCIe.[51]
Linux
Intel published an NVM Express driver for Linux on 3 March 2011,[52][53][54] which was merged into the Linux kernel mainline on 18 January 2012 and released as part of version 3.3 of the Linux kernel on 19 March 2012.[55]
NetBSD
NetBSD added support for NVMe in NetBSD 8.0.[56] The implementation is derived from OpenBSD 6.0.
OpenBSD
Development work required to support NVMe in OpenBSD has been started in April 2014 by a senior developer formerly responsible for USB 2.0 and AHCI support.[57] Support for NVMe has been enabled in the OpenBSD 6.0 release.[58]
OS X/macOS
In the 10.10.3 update for OS X Yosemite, Apple introduced support for NVM Express. The Retina MacBook and 2016 MacBook Pro use NVMe over PCIe as the logical-device interface.[59]
Solaris
Solaris received support for NVMe in Oracle Solaris 11.2.[60]
VMware
Intel has provided an NVMe driver for VMware,[61] which is included in vSphere 6.0 and later builds, supporting various NVMe devices.[62] As of vSphere 6 update 1, VMware's VSAN software-defined storage subsystem also supports NVMe devices.[63]
Windows
Microsoft added native support for NVMe to Windows 8.1 and Windows Server 2012 R2.[37][64] Native drivers for Windows 7 and Windows Server 2008 R2 have been added in updates.[65]
The OpenFabrics Alliance maintains an open-source NVMe Windows Driver for Windows 7/8/8.1 and Windows Server 2008R2/2012/2012R2, developed from the baseline code submitted by several promoter companies in the NVMe workgroup, specifically IDT, Intel, and LSI.[66] The current release is 1.5 from December 2016.[67]

Software support
QEMU
NVMe is supported by QEMU since version 1.6 released on August 15, 2013.[68]
UEFI
An open source NVMe driver for UEFI is available on SourceForge.[69]

Management tools

nvmecontrol[70]

The nvmecontrol tool is used to control an NVMe disk from the command line on FreeBSD. It was added in FreeBSD 9.2.

nvme-cli[71]

NVM-Express user space tooling for Linux.

gollark: As I said, it makes the control flow seem backwards.
gollark: I dislike your use of after.
gollark: `.doAThingButOnlyAfterExecutionOfThePreviousThingHasSuccessfullyCompleted`
gollark: `afterThePreviousThingDoTheNextThing`1
gollark: `.andThen`

See also

References
  1. Compare: "NVM Express". NVM Express, Inc. Retrieved 2017-01-24. NVMe™ is designed from the ground up to deliver high bandwidth and low latency storage access for current and future NVM technologies.
  2. Drew Riley (2014-08-13). "Intel SSD DC P3700 800GB and 1.6TB Review: The Future of Storage". tomshardware.com. Retrieved 2014-11-21.
  3. "Intel Solid-State Drive DC P3600 Series" (PDF). Intel. 2015-03-20. pp. 18, 20–22. Retrieved 2015-04-11.
  4. Paul Alcorn (2015-06-05). "SFFWG Renames PCIe SSD SFF-8639 Connector To U.2". Tom's Hardware. Retrieved 2015-06-09.
  5. Dave Landsman (2013-08-09). "AHCI and NVMe as Interfaces for SATA Express Devices – Overview" (PDF). SATA-IO. Retrieved 2013-10-02.
  6. Paul Wassenberg (2013-06-25). "SATA Express: PCIe Client Storage" (PDF). SATA-IO. Retrieved 2014-11-21.
  7. NVMe Specifications
  8. Walker, Don H. "A Comparison of NVMe and AHCI" (PDF). 31 July 2012. SATA-IO. Retrieved 3 July 2013.
  9. "NVM Express Explained" (PDF). nvmexpress.org. 9 April 2014. Retrieved 21 March 2015.
  10. "Using LC's Sierra Systems". hpc.llnl.gov. Retrieved 2020-06-25.
  11. "SummitDev User Guide". olcf.ornl.gov. Retrieved 2020-06-25.
  12. "Speeding up Flash... in a flash". The Inquirer. 2007-10-13. Retrieved 2014-01-11.
  13. http://www.bswd.com/FMS09/FMS09-T2A-Huffman.pdf
  14. "Flash new standard tips up". The Inquirer. 2008-04-16. Retrieved 2014-01-11.
  15. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2008/20080813_T2A_Huffman.pdf
  16. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130813_A12_Onufryk.pdf
  17. "New Promoter Group Formed to Advance NVM Express" (PDF). Press release. June 1, 2011. Retrieved September 18, 2013.
  18. Amber Huffman editor (October 11, 2012). "NVM Express Revision 1.1" (PDF). Specification. Retrieved September 18, 2013.
  19. David A. Deming (2013-06-08). "PCIe-based Storage" (PDF). snia.org. Archived from the original (PDF) on 2013-09-20. Retrieved 2014-01-12.
  20. Amber Huffman editor (January 23, 2013). "NVM Express Revision 1.0e" (PDF). Specification. Retrieved September 18, 2013.
  21. "IDT releases two NVMe PCI-Express SSD controllers". The Inquirer. 2012-08-21. Retrieved 2014-01-11.
  22. "IDT Shows Off The First NVMe PCIe SSD Processor and Reference Design - FMS 2012 Update". The SSD Review. 2012-08-24. Retrieved 2014-01-11.
  23. "Samsung Announces Industry's First 2.5-inch NVMe SSD | StorageReview.com - Storage Reviews". StorageReview.com. 2013-07-18. Archived from the original on 2014-01-10. Retrieved 2014-01-11.
  24. "LSI SF3700 SandForce Flash Controller Line Unveiled | StorageReview.com - Storage Reviews". StorageReview.com. 2013-11-18. Archived from the original on 2014-01-11. Retrieved 2014-01-11.
  25. "LSI Introduces Blazing Fast SF3700 Series SSD Controller, Supports Both PCIe and SATA 6Gbps". hothardware.com. Retrieved 21 March 2015.
  26. Jane McEntegart. "Kingston Unveils First PCIe SSD: 1800 MB/s Read Speeds". Tom's Hardware. Retrieved 21 March 2015.
  27. "Kingston HyperX Predator PCI Express SSD Unveiled With LSI SandForce SF3700 PCIe Flash Controller". hothardware.com. Retrieved 21 March 2015.
  28. "Intel® Solid-State Drive Data Center Family for PCIe*". Intel. Retrieved 21 March 2015.
  29. "NVM Express » NVM Express Organization History". nvmexpress.org. Archived from the original on 23 November 2015. Retrieved 23 December 2015.
  30. "PowerEdge R730xd Rack Server". Dell. Retrieved 2019-03-29.
  31. "Fibre Channel Industry Association Collaborating with NVM Express, Inc. to Define New Standard for Fabric-Based Solid-State Storage Devices" (Press release).
  32. "NVM Express over Fabrics Revision 1.0" (PDF). NVM Express, Inc. 5 June 2016.
  33. Woolf, David (9 February 2018). "What NVMe over Fabrics Means for Data Storage".
  34. Hellwig, Christoph (17 July 2016). "NVMe Over Fabrics Support in Linux" (PDF).
  35. Stern, Jonathan (7 June 2016). "Announcing the SPDK NVMf Target".
  36. Randall, Robert (20 July 2016). "Windows NVMeOF host / initiator demo at FMS".
  37. Andy Herron (2013). "Advancements in Storage and File Systems in Windows 8.1" (PDF). snia.org. Archived from the original (PDF) on 2014-01-10. Retrieved 2014-01-11.
  38. Amber Huffman (March 9, 2020). "NVM Express Base Specification Revision 1.4a" (PDF). Specification. section 1.4 Theory of Operation, p. 7. Retrieved May 16, 2020.
  39. Werner Fischer; Georg Schönberger (2015-06-01). "Linux Storage Stack Diagram". Thomas-Krenn.AG. Retrieved 2015-06-08.
  40. "NVM Express » ChromeOS adds boot support for NVM Express". nvmexpress.org. Retrieved 21 March 2015.
  41. "4f503189f7339c667b045ab80a949964ecbaf93e - chromiumos/platform/depthcharge - Git at Google". googlesource.com. Retrieved 21 March 2015.
  42. "DragonFly BSD 4.6". www.dragonflybsd.org. Retrieved 2016-09-08.
  43. "Log of /head/sys/dev/nvme". FreeBSD source tree. The FreeBSD Project. Retrieved 16 October 2012.
  44. "Log of /stable/9/sys/dev/nvme". FreeBSD source tree. The FreeBSD Project. Retrieved 3 July 2013.
  45. "FreeBSD 10.2-RELEASE Release Notes". The FreeBSD Project. Retrieved 5 August 2015.
  46. "Release notes for the Genode OS Framework 18.05". genode.org.
  47. "#9910 NVMe devices support". dev.haiku-os.org. Retrieved 2019-04-18.
  48. "NVMe Driver Now Available - Haiku Project". www.haiku-os.org. Retrieved 2016-07-28.
  49. "4053 Add NVME Driver Support to Illumos". github.com. Retrieved 2016-05-23.
  50. Ho, Joshua (September 28, 2015). "iPhone 6s and iPhone 6s Plus Preliminary Results". AnandTech. Retrieved 2016-06-01.
  51. Chester, Brandon (May 16, 2016). "The iPhone SE Review". AnandTech.
  52. Matthew Wilcox (2011-03-03). "NVM Express driver". LWN.net. Archived from the original on 2012-07-17. Retrieved 2013-11-05.
  53. Keith Busch (2013-08-12). "Linux NVMe Driver" (PDF). flashmemorysummit.com. Retrieved 2013-11-05.
  54. "IDF13 Hands-on Lab: Compiling the NVM Express Linux Open Source Driver and SSD Linux Benchmarks and Optimizations" (PDF). activeevents.com. 2013. Archived from the original (PDF) on 2014-01-11. Retrieved 2014-01-11.
  55. "Merge git://git.infradead.org/users/willy/linux-nvme". kernel.org. 2012-01-18. Retrieved 2013-11-05.
  56. "nvme -- Non-Volatile Memory Host Controller Interface". NetBSD manual pages. 2016-01-01. Retrieved 2016-09-02.
  57. David Gwynne (2014-04-16). "non volatile memory express controller (/sys/dev/ic/nvme.c)". BSD Cross Reference. Retrieved 2014-04-27.
  58. David Gwynne (2016-04-14). "man 4 nvme". OpenBSD man page. Retrieved 2016-08-07.
  59. "Faster 'NVM Express' SSD Interface Arrives on Retina MacBook and OS X 10.10.3". macrumors.com. Retrieved 11 April 2015.
  60. "nvme(7D)". Oracle. Retrieved 2014-12-02.
  61. "Intel Solid-State for NVMe Drivers". intel.com. 2015-09-25. Retrieved 2016-03-17.
  62. "VMware Compatibility Guide for NVMe devices". vmware.com. Retrieved 2016-03-17.
  63. "VSAN Now Supporting NVMe Devices". vmware.com. 2015-11-11. Retrieved 2016-03-17.
  64. "Windows 8.1 to support hybrid disks and adds native NVMe driver". Myce.com. 2013-09-06. Retrieved 2014-01-11.
  65. "Update to support NVM Express by using native drivers in Windows 7 or Windows Server 2008 R2". Microsoft. 2014-11-13. Retrieved 2014-11-17.
  66. "Windows NVM Express". Project web site. Archived from the original on June 12, 2013. Retrieved September 18, 2013.
  67. https://svn.openfabrics.org/svnrepo/nvmewin/releases/
  68. "ChangeLog/1.6". qemu.org. Retrieved 21 March 2015.
  69. "Download EDK II from". SourceForge.net. Retrieved 2014-01-11.
  70. NVM Express control utility, The FreeBSD Project, 2018-03-12, retrieved 2019-07-12
  71. GitHub - linux-nvme/nvme-cli: NVMe management command line interface., linux-nvme, 2019-03-26, retrieved 2019-03-27
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.