3

I noticed that writing to raid-6 is very low, but when I make tests with hdparm the speed is reasonable: 

dd if=/dev/zero of=/store/01/test.tmp bs=1M count=10000

Gives: 50Mb/s or even less.

The hdparm gives: hdparm --direct -t /dev/vg_store_01/logical_vg_store_01 Gives 450MB/s

Why the file writings are low than hdparm test? Are there some kernel limit should be tuned?

I have an Areca 1680 adapter with 16x1Tb SAS disks, scientific linux 6.0

EDIT

My bad. Sorry all units are in MB/s

More on hardware:

2 areca contollers in dual quadcore machine. 16Gb ram
the firmware for sas backplane and areca is recent one.
the disks are seagate 7.200 rpm 16x1Tb x2 raid boxes. each 8 disks are raid6, so total 4 volumes with lba=64.

two volumes groupped by striped lvm and formatted ext4

the stripe size is 128

when I format the volume I can see by iotop it writes 400mb/s

iostat shows also that both lvm member drives are writing with 450MB/s

FINALLY WRITING with 1600GB/s

One of the raids was degrading the performance due to bad disk. It is strange that disk in the jbod mode gives 100MB/s with hdparm as others. After heavy IO, it was reporting in the log files Write Error(not it has 10 of them). The raid still was not failing or degrading.

Well after replacement my configuration is following:

  • 2xARC1680 controllers with
  • RAID0 with 16x1Tb SAS disks stripe 128 lba64
  • RAID0 with 16x1Tb SAS disks stripe 128 lba64

volume group with 128K stripe size

formatted to XFS

Direct

hdparm --direct -t /dev/vg_store01/vg_logical_store01

/dev/vg_store01/vg_logical_store01: Timing O_DIRECT disk reads: 4910 MB in 3.00 seconds = 1636.13 MB/sec

No Direct

hdparm -t /dev/vg_store01/vg_logical_store01

/dev/vg_store01/vg_logical_store01: Timing buffered disk reads: 1648 MB in 3.00 seconds = 548.94 MB/sec

** dd test DIRECT**

dd if=/dev/zero of=/store/01/test.tmp bs=1M count=10000 oflag=direct 10000+0 records in 10000+0 records out 10485760000 bytes (10 GB) copied, 8.87402 s, 1.2 GB/s

** WITHOUT DIRECT**

dd if=/dev/zero of=/store/01/test.tmp bs=1M count=10000 10000+0 records in 10000+0 records out 10485760000 bytes (10 GB) copied, 19.1996 s, 546 MB/s

Arman
  • 555
  • 2
  • 8
  • 18

5 Answers5

3

Check if your FS is aligned with RAID dimensions. I'm getting 320MB/s on RAID-6 array with 8 x 2TB SATA drives on XFS and I think it is limited by 3Gb/s SAS channel rater then RAID-6 performance. You can get some ideas on alignment from this thread.

Community
  • 1
dtoubelis
  • 4,579
  • 1
  • 28
  • 31
2

Unfortunately you're comparing apples with oranges.

450Mb/s = 56MB/s which is about on par with what you're seeing in real life. They're both giving you the same reading (but one is in bits, one is in bytes). You need to divide 450 by 8 to get the same measure for both.

(In your question, you've got the capitalisation the other way around, now I can only hope/assume that this is a typo error, because if you reverse the capitalisation you get an almost perfect match)

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
  • Unfortunately™, you're deeply wrong. There's no reason for hdparm to speak in terms of bits at all, so it doesn't. It uses "MebiBytes" per second and you can check it out by yourself. – poige Apr 03 '11 at 00:07
  • 3
    @Poige - you'll have to forgive me then, I was simply working off the fact that the op talked in Mbs and then MBs, of which there is a large difference. – Mark Henderson Apr 03 '11 at 04:56
  • Sorry my mistake all units are in MB/s – Arman Apr 04 '11 at 07:17
0

Check if you can enable write cache on the raid controller

its best if you have battery on your controller otherwise you may lose data during power failure

John
  • 391
  • 1
  • 4
0

Doesn't the hdparm test basically perform buffered read tests? You can't compared buffered read speeds to actually write speeds and expected them to be equal on a RAID6 device.

While I would expect better performance than 50MB/sec writes on RAID6 of that size with quality drives (1TB SAS or 1TB SATA?), I wouldn't expect 450MB/sec write speeds.

rthomson
  • 1,059
  • 9
  • 14
0

but when I make tests with hdparm the speed is reasonable

hdparm does not test write performance, it's "read-only". Moreover, it tests actually block I/O read performance, but the way you invoke dd makes it test both write and filesystem performance as well (and RAID-5,6 write is noticeable slower than read by design). If your FS is EXT3, for e. g., you can easily get poor performance having not formatted it properly (taking not into consideration full stripe size parameter of your RAID).

Also, there's quite a big number of people who tend to use rather small stripe sizes which leads to suboptimal disk I/O. What was your stripe size choice when creating this RAID?

Another question is how dd's numbers differ while varying bs parameter? Have you tried using full stripe write size for it?

poige
  • 9,171
  • 2
  • 24
  • 50