3

I've just built a small NAS/SAN box and am doing some testing with it. Configuration is:

  • 2GB RAM
  • 2x Opteron 265s
  • Iwill DK8N
  • 5x500GB SATA drives (1 spare, rest mirrored) = 1TB usable space
  • Emulex LPC10000DC
  • NexentaCore 3.0.2

The first thing I'm testing is COMSTAR performance over fibre-channel. I created a 700g zvol and exposed it over COMSTAR. Write-caching is enabled. My FC initiator is a Windows 7 box with a Q-Logic QLA2340 card in it.

Everything works great, except for write speeds. For some reason, I'm only seeing ~30MB/s, whereas I can peak at about 90MB/s read speeds, using IOMeter. Doing some hunting around, I ran a zpool iostat command and saw the following pattern:

$ zpool iostat tank 1
tank        88.2G   840G      0  5.47K      0  43.4M
tank        88.2G   840G      0  16.1K      0   128M
tank        88.4G   840G      1  6.69K  8.93K  52.6M
tank        88.4G   840G     52  4.89K   110K  38.3M
tank        88.4G   840G    151      0   280K      0
tank        88.4G   840G    112      0   253K      0
tank        88.4G   840G      1  9.89K  16.0K  78.6M
tank        88.4G   840G      0  14.3K      0   113M
tank        88.6G   839G    150  2.99K   318K  20.2M
tank        88.6G   839G     15      0  45.0K      0
tank        88.6G   839G      3      0  17.8K      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0  11.3K      0  89.9M
tank        88.6G   839G      0  13.8K      0   110M
tank        88.8G   839G     90  7.91K   184K  61.0M
tank        88.9G   839G    105     91   220K   140K
tank        88.9G   839G     42      0  84.6K      0
tank        88.9G   839G      0      0      0      0
tank        88.9G   839G      0  6.52K  7.99K  51.8M
tank        88.9G   839G      0  15.6K   1011   124M
tank        88.9G   839G     91  4.30K   193K  31.9M
tank        89.1G   839G      1  9.48K  15.8K  74.7M
tank        89.1G   839G     41      0  99.9K      0
tank        89.1G   839G    166      0   374K      0
tank        89.1G   839G     87      0   162K      0
tank        89.1G   839G      0      0  7.92K      0

It appears to be writing a lot, then stopping.

What's going on here?

growse
  • 7,830
  • 11
  • 72
  • 114

1 Answers1

1

That's normal for ZFS. What's happening is it's coalescing the writes in memory and then committing them to disk at the same time. This a great wiki about what it's doing and how you can tune it:

http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

JOTN
  • 1,727
  • 1
  • 10
  • 12
  • Interesting. I guess my concern was around the speed seen by the COMSTAR initiator - Bonnie++ and these results indicate that 4 spindles can do ~125M/s quite easily, seeing only 30MB/s at the other end of a 2GB FC link made me question what I was doing wrong. Also, the fact that I see a similar bursting pattern with IOMeter on the client gave cause for concern. If it were caching properly, I'd expect to see a fairly constant IO level on the client. I'll read up on that link and see if I can get total throughput up anyhow. – growse Feb 11 '11 at 09:13