0
Disks: 10xSSD Ultrastar SS200 960GB 12GB/s Raid 0, 6, 10.
Controller: LSI Syncro 9380-8e
Filesystem: ext4 without LVM
System: Centos 7
2x E5620  @ 2.40GHz
32GB RAM
fio-2.1.10: --iodepth=32 --ioengine=libaio --rw=randrw --bs=4k --size 10000M -numjobs=10

At the beginning of testing i have aroud 60k IOPS in raid 0, after 2-3 minutes the counter falls to 2-5k IOPS

Start:

 Jobs: 10 (f=10): [mmmmmmmmmm] [10.6% done] [123.6MB/123.6MB/0KB /s] [31.7K/31.7K/0 iops] [eta 13m:40s]

After:

 Jobs: 10 (f=10): [mmmmmmmmmm] [14.5% done] [4839KB/4723KB/0KB /s] [1209/1180/0 iops] [eta 14m:18s]

Top in this moment:

 top - 09:19:06 up  4:45,  2 users,  load average: 10.41, 5.78, 2.41
 Tasks: 282 total,   2 running, 280 sleeping,   0 stopped,   0 zombie
 %Cpu(s):  0.6 us,  3.5 sy,  0.0 ni, 42.8 id, 52.9 wa,  0.0 hi,  0.3 si,       0.0 st
 KiB Mem : 32769764 total,   214292 free,   334168 used, 32221304 buff/cache
 KiB Swap: 16515068 total, 16515068 free,        0 used. 31963788 avail Mem

I think it is the low perfomance for 10 ssd (60K iops rw) even every disk can handle 30-40 iops.

I tried 2 different controllers, 3 types of raid, windows and linux systems - i have the same result of testing. What is my problem? How to understand why perfomance is too low and why i have huge perfomance drops? I hear about SSD spare area, but i still do not understand how to reconfigure it.

shallrise
  • 85
  • 1
  • 13
  • How is the ext4 partition mounted? – Jaroslav Kucera Nov 08 '17 at 14:58
  • mount /dev/sdb /mnt – shallrise Nov 08 '17 at 15:01
  • I mean mount params from /etc/fstab... Is there the latest firmware? – Jaroslav Kucera Nov 08 '17 at 15:01
  • I do not use fstab, just mount logical volume by command line. Can you tell what parameters I should use in fstab and i will do that. – shallrise Nov 08 '17 at 15:08
  • OK, try to remount it with noatime. And as you use RAID 0 anyway, you can use nobarrier. Also in case the RAID controller has battery backup (of flash backup) cache, then nobarrier is safe. You can pass those params from mount command too (with -o noatime,nobarrier). – Jaroslav Kucera Nov 08 '17 at 15:14
  • Also these disks are considered "read intensive". So the most probably the SLC cache area of these disks is probably quite small. Try readread mode instead of readrw. – Jaroslav Kucera Nov 08 '17 at 15:25
  • What raid controllers did you use? With how much non volatile writeback cache? – shodanshok Nov 08 '17 at 16:29
  • Jaroslav - With nobarrier,noatime i have the same situation. If i add direct=1 fio parametr i have [11.5K/11.9K/0 iops] without drops on 4 raid 0 ssd. shodanshok - LSI Syncro 9380-8e, also i test with i410.Cant say about writeback right now. – shallrise Nov 08 '17 at 16:52

1 Answers1

1

That fio job is allowed to do buffered writes and as such the write data can be buffered in RAM and only sent to disk (via writeback) at a later point. Given this, one explanation could be that initially most of the I/O is going into RAM but then at some point the RAM buffer starts being flushed and data actually has to be sent to the backing store causing new I/O into the buffer to have to wait leading to a slower speed.

Also note that as the fio documentation points out the libaio ioengine may not be asynchronous when using buffered I/O mode AND you're using it to do I/O into an ext4 filesystem (which again can invoke more blocking).

Finally are you sure you controller supports an iodepth of 320 (32 x 10) - it might do but it's worth checking...

TLDR; the fio job's results may be being massively distorted by buffering and might not be achieving the I/O depths expected.

Anon
  • 1,210
  • 10
  • 23