1

Our server has a raid6 array (10+2 disks, each disk is 6TB/SATA/7200rpm). We plan to make it an NFS storage shared with other clients. However, the writing speed is very slow:

[root@Server ~]# mkfs.xfs /dev/sdd1
meta-data=/dev/sdd1              isize=512    agcount=55, agsize=268435455 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=14640739328, imaxpct=1
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@Server ~]# mount /dev/sdd1 /share/
[root@Server ~]# mkdir -p /share/aa/bb/cc/dd/ee
[root@Server ~]# cd /share/aa/bb/cc/dd/ee
[root@Server ~]# dd if=/dev/zero of=tempfile bs=1M count=30000 conv=fsync
30000+0 records in
30000+0 records out
31457280000 bytes (31 GB) copied, 1186.26 s, 26.5 MB/s
[root@Server ~]# cd /share
[root@Server ~]# dd if=/dev/zero of=tempfile bs=1M count=30000 conv=fsync
30000+0 records in
30000+0 records out
31457280000 bytes (31 GB) copied, 109.495 s, 287 MB/s

It is weird that writing to /share is much faster (287MB/s) than writing to deeper subfolders (26MB/s). I tested for many times, and the results were similar.

I also tried formating it to other file systems, such as ext4 and brtfs, but both of them behaved normally in subfolders (592MB/s and 226MB/s).

ext4:

[root@Server ~]# mkfs.ext4 /dev/sdd1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
915046400 inodes, 14640739328 blocks
732036966 blocks (5.00%) reserved for the super user
First data block=0
446800 block groups
32768 blocks per group, 32768 fragments per group
2048 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544, 1934917632, 
    2560000000, 3855122432, 5804752896, 12800000000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done         

[root@Server ~]# mount /dev/sdd1 /share/
[root@Server ~]# mkdir -p /share/aa/bb/cc/dd/ee
[root@Server ~]# cd /share/aa/bb/cc/dd/ee
[root@Server ~]# dd if=/dev/zero of=tempfile bs=1M count=30000 conv=fsync
30000+0 records in
30000+0 records out
31457280000 bytes (31 GB) copied, 53.1054 s, 592 MB/s

btrfs:

[root@Server ~]# mkfs.btrfs /dev/sdd1
btrfs-progs v4.9.1
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               32aa3677-7898-4f14-9158-b698bd11abbc
Node size:          16384
Sector size:        4096
Filesystem size:    54.54TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    54.54TiB  /dev/sdd1

[root@Server ~]# mount /dev/sdd1 /share/
[root@Server ~]# mkdir -p /share/aa/bb/cc/dd/ee
[root@Server ~]# cd /share/aa/bb/cc/dd/ee
[root@Server ~]# dd if=/dev/zero of=tempfile bs=1M count=30000 conv=fsync
30000+0 records in
30000+0 records out
31457280000 bytes (31 GB) copied, 139.226 s, 226 MB/s

I googled for several weeks, but found no solution. Could anyone here give me some hints?

PS: the OS is CentOS 7.4

xlwang
  • 11
  • 2
  • Please share more information on your RAID array - specifically, RAID stripe size and RAID card model/version. – shodanshok Feb 10 '19 at 00:29
  • Thanks! When checking the RAID information in the BIOS, I found the write cache is disabled by default. After enabling it, the writing speed is 1.x GB/s now. BTW: the RAID card is PM8060, and the stripe size is 256k. – xlwang Feb 15 '19 at 05:31

0 Answers0