We are testing the speed of two Intel DC P3600 NVMe drives on a CentOS 7.1 system. They are configured in a software RAID1 using mdadm.
After upgrading the kernel from the (very old) 3.10 to the latest 4.3 (or 4.2.5) we see can observe a very dramatic read speed degradation - from 4GB/s to max 500MB/s. We can replicate the test by selecting the kernel version at boot time.
The tests are conducted using fio with using the following command:
fio --name=read --ioengine=libaio --iodepth=16 --rw=read \
--bs=128k --direct=1 --size=32G --numjobs=8 --runtime=300 \
--time_based=1 --group_reporting
What change in the Kernel could cause this? Everywhere I read, the upgrade is even recommended as the NVMe driver reached version 1.0 after kernel 3.19. Any suggestions would be more than welcome.