ZFS not writing continuously to array

2

Preface: I'm by no means an expert Linux user, but I've played with Linux enough to be dangerous.

I'm building a personal server, basically just to use as storage for video and photos. Current specs are as follows:

  • Gigabyte GA-G41M-ES2L Motherboard
  • Core 2 Duo E6750 (2.66GHz) CPU
  • 4GB DDR2 RAM
  • Vantec UGT-ST644R PCIe SATA Card (no RAID being used)
  • 2 × 1.5TB and 2 × 1.0TB Hard Drives
  • 64GB SSD (for boot drive)

I have installed Ubuntu Server 14.04.3 LTS to the SSD, with ZFSonLinux installed as per here (Ars Technica). Also installed is Samba (Version 4.1.6-Ubuntu), which I'm using to access the server from the rest of my (Windows 7 and up) computers (two or three, at most, eventually).

All four drives are connected to the Vantec controller card, and each pair is mirrored and added to the ZPool, giving a total of 2.5TB of space. The SSD is connected directly to the on-board controller.

When it works, it works beautifully, however...

When I write large amounts of data to the array, the first ≈500MB "write" just fine (to the cache, I assume), then ZFS waits around for ≈30 seconds, (monitoring using zpool iostat <pool_name> 1), writes ≈350MB out of cache to the array while accepting more data in, waits ≈30 seconds, writes ≈350MB out of cache to the array while accepting more data in, waits ≈30 seconds, etc., until the transfer is finished. In those 30 second chunks, it's doing nothing (0 in, 0 out), just sitting there idle.

The network link is (much to my surprise, actually) capable of filling the 350MB in about 5 seconds, leaving a lot of dead time during the whole process.

Is there a way to remove this bottleneck / 'breathing'?

I've tried changing the zfs_txg_timeout value (default is 5 seconds, so that seems unlikely, but 1 second doesn't seem to help).

I've also set the min and max cache values to 2GB and 3GB respectively, but it still starts with a relatively small cache and slowly grows. I've watched the system's memory used during this, and ZFS seems reluctant to rapidly increase it's RAM use, even when everything is waiting on it.

I like the way that ZFS allows me to stripe across different size vdevs, but the described behavior makes it next-to-unusable for transferring large amounts of data to it reliably.

Any suggestions, either to fix the current setup, or of an alternative that would work comparably? Thank you in advance for the insight. If you need any more information, let me know.

Side Notes:

  1. I've run a local disk benchmark on the array, and the same behavior presented itself there, as well, ruling out both the Windows computers and the network.
  2. Running sync seems to do some good, but not consistently nor permanently.
  3. Memory is maxed out for this configuration, but I don't think that it's immediately the issue at hand.
  4. My hope is that this isn't a controller issue (I don't have another one, and the board only has four ports (one for the SSD, three remaining)).
  5. Most sources seem to say that a ZIL disk won't help in my situation.

Updates (Sept 1, 2015):

  1. I installed FreeNAS on the same hardware, using the same array topology, and ran into the exact same issue. This mean the problem is most likely hardware.
  2. To that end, I put three of the four drives on the internal controller and built an array out of that. Problem solved! Continuous writes achieved. Granted, it means I need to find a new SATA card, but at least it works in theory now.

MobiusOne

Posted 2015-09-01T03:26:31.097

Reputation: 21

2Hi MobiusOne. Please don't edit your question to answer it. Instead, post the solution as an answer of your own (a self-answer) and accept that. That will tell the community that you feel your question has been answered. It looks like your most recent edit should do quite acceptably as a self-answer, if only reworded slightly to read more like an answer. – a CVn – 2015-09-02T19:32:34.617

And no, a ZIL most likely won't help. Technically that's a SLOG device (there is always a ZIL on a ZFS pool, even if it isn't kept on a separate storage device), but even so, it only helps with synchronous writes, which is a very small part of a normal workstation-like I/O workflow. – a CVn – 2015-09-02T19:33:33.973

For this problem (and other similar ones) the first thing you want to do is look at the output of iostat -x to see where the actual bottleneck(s) lie. – qasdfdsaq – 2015-10-18T00:16:56.003

No answers