2

yesterday I added new harddrives(four as a raidz1 and one as hot-spare) to a opensolaris server, after extending the zpool the server hangs when writing large files but not when reading large files(large files = > 1GiB).

The zpool configuration before the upgrade looked like this:

state: ONLINE

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t0d0 ONLINE 0 0 0
  c9t1d0 ONLINE 0 0 0
  c9t2d0 ONLINE 0 0 0
  c9t3d0 ONLINE 0 0 0

After the upgrade the zpool looks like this:

state: ONLINE

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t0d0 ONLINE 0 0 0
  c9t1d0 ONLINE 0 0 0
  c9t2d0 ONLINE 0 0 0
  c9t3d0 ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
  c9t4d0 ONLINE 0 0 0
  c9t5d0 ONLINE 0 0 0
  c9t6d0 ONLINE 0 0 0
  c9t7d0 ONLINE 0 0 0
 spares
  c9t8d0 AVAIL

As you can see all drives are Online an even the 3Ware 9690SA-4I Controller tells me that everything is okey:

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
----------------------------------------------------------------------------- -
u0 SINGLE OK - - - 1862.63 RiW ON
u1 SINGLE OK - - - 1862.63 RiW ON
u2 SINGLE OK - - - 1862.63 RiW ON
u3 SINGLE OK - - - 1862.63 RiW ON
u4 SINGLE OK - - - 1862.63 RiW ON
u5 SINGLE OK - - - 1862.63 RiW ON
u6 SINGLE OK - - - 1862.63 RiW ON
u7 SINGLE OK - - - 1862.63 RiW ON
u8 SINGLE OK - - - 1862.63 RiW ON

VPort Status Unit Size Type Phy Encl-Slot Model
----------------------------------------------------------------------------- -
p8 OK u0 1.82 TB SATA - /c9/e0/slt1 SAMSUNG HD203WI
p9 OK u1 1.82 TB SATA - /c9/e0/slt3 SAMSUNG HD203WI
p10 OK u2 1.82 TB SATA - /c9/e0/slt5 SAMSUNG HD203WI
p11 OK u4 1.82 TB SATA - /c9/e0/slt6 SAMSUNG HD203WI
p12 OK u5 1.82 TB SATA - /c9/e0/slt8 SAMSUNG HD203WI
p13 OK u3 1.82 TB SATA - /c9/e0/slt10 SAMSUNG HD203WI
p14 OK u6 1.82 TB SATA - /c9/e0/slt13 SAMSUNG HD203WI
p15 OK u7 1.82 TB SATA - /c9/e0/slt15 SAMSUNG HD203WI
p16 OK u8 1.82 TB SATA - /c9/e0/slt17 SAMSUNG HD203WI

But when I start writing Files to this zfs the server hangs sometime during the write process and sometimes just after writing the whole file but for sure the server hangs... . Reading large files(7-8GiB) on the otherside is no problem!

Thanks for your answers!

cu

Guido

edit:

fyi: The server runs at svn_111b

edit 2:

scrub: scrub completed after 6h20m with 0 errors on Thu Jul 22 00:33:29 2010

As you can see there are no file system errors... .

user48937
  • 23
  • 5
  • Do you see any particular in /var/adm/messages? When the write hangs, if you do a "zfs list -t all" what do you get? – PiL Jul 22 '10 at 08:12
  • storage 3,85T 6,83T 29,9K /storage storage/share 3,85T 6,83T 3,85T /storage/share But it seems I found the reason the 3Ware 9690SA-4I controller produced timeouts because write caching was enabeld(by default) and no BBU is available wich let the controller gone mad after adding the new hdds... – user48937 Jul 22 '10 at 09:41
  • It's always better disable hardware cache on a ZFS system. Especially when no BBU is present. – PiL Jul 22 '10 at 10:21
  • But the server still crashes... /var/adm/messages : Jul 22 18:19:00 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 3 satapkt 0xffffff019bdfd510 timed out Jul 22 18:23:20 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff019846d0b8 timed out Jul 22 18:39:35 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff01aa17c6c8 timed out Jul 22 18:51:31 sodom ahci: [ID 517647 kern.warning] WARNING: ahci0: watchdog port 2 satapkt 0xffffff019e86ba50 timed out – user48937 Jul 23 '10 at 06:36
  • You said you have a 3ware...once i had problems with it, especially if it's associated with a particular board. Which server do you have? You run a pretty old version of opensolaris...have you tried to upgrade? – PiL Jul 23 '10 at 07:58
  • Thanks for your help! The Mainboard is a Supermicro X8DT3-F. Do you think i should upgrade to a developent version like snv_136? – user48937 Jul 23 '10 at 09:22
  • I had a supermicro too and i had a problem with a 3ware card (some compatibility problem with pci-x). As far as i read there were some problems related to some sata timeout in that version. You could try to update an alternate boot environment, so you can always go back to the original conf. – PiL Jul 23 '10 at 10:04
  • thx i will try it ;) – user48937 Jul 23 '10 at 10:13
  • Hi caco: Are you using 3ware LSI 9650 raid Card? please help as i am considering buying a new one – JMS77 Aug 03 '10 at 07:16

1 Answers1

-2

It's a 3+ years bug with ZFS ARC that still persists!

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017

It will also go out-of-bounds from the VM limits of a hypervisor!

user48838
  • 7,393
  • 2
  • 17
  • 14
  • 1
    No, it isn't. Please stop assuming that this bug is relevant to every ZFS issue and/or question people post. – EEAA Jul 26 '10 at 02:20
  • Have you had direct experience with this bug as it crashes the OS into a spontaneous reboot and then find that adjusting the ZFS ARC setting actually reduces this exact effect? What are you basing your conclusions of "No, it isn't." to??? – user48838 Jul 26 '10 at 02:42
  • 1
    No I haven't, but it's pretty easy to determine from the text in the bug report that it has nothing to do with crashing *or* poor performance. Perhaps you linked to the wrong bug report? – EEAA Jul 26 '10 at 02:58
  • I've been battling with this exact bug and saw first-hand that it is not isolated/limited to the kernel when in a virtualized environment. I saw it leaking into Physical Memory of the host, which may be a hypervisor-related situation, as stated in the the second paragraph of the bug's description. – user48838 Jul 26 '10 at 03:07
  • 1
    What makes you think this person is running ZFS in a VM environment? – EEAA Jul 26 '10 at 03:09
  • I didn't, if you read my comment(s) carefully, I was able to verify that this bug still exists in the wild while I was running in a virtulaized environment. The same symptoms materialized when my ZFS store grew to the point in which this bug was triggered. I was just fortunate enough to spot it (and head off some of its negative effects) - largely due to its bad behavior in a virtulized environment. Do you have a vested interest debunking the existence of this condition? What is your ZFS "rig" configuration for comparison? – user48838 Jul 26 '10 at 03:15
  • An easy tie-breaker is for caco (OP) to see if adjusting their ZFS ARC setting improves their situation. – user48838 Jul 26 '10 at 03:26
  • 1
    No, the only vested interest I have is in ensuring the accuracy of answers on SF. Up until this point, I've seen *zero* evidence that the bug you posted has anything to do with any of the questions you answered. – EEAA Jul 26 '10 at 03:36
  • Once again... What is your ZFS "rig" for comparison and how about letting caco make the actual determination since I am actually offering a possible solution/stop-gap via our interchanges? – user48838 Jul 26 '10 at 03:40
  • 1
    @user48838 at the moment it seems that resizing the ARC had solved my problem(I resized it to 1,5GiB from 2,4GiB[default]), this morning I could copy 10 DVD images to the server without a crash. The performance has declined but its still better than a crashing server :D ... Do you thing adding more memory would let me extend the ARC size again? – user48937 Jul 26 '10 at 07:57
  • 50GiB later the Server is still running... – user48937 Jul 26 '10 at 10:26
  • Unfortunately not from my testing. Another possible positive "compounding" setting may be to increase the ZFS "block size" which may reduce the ZFS ARC entries, a possible trade-off of efficiency if you store a lot of smaller files in your file system. I still need to test that... caco, not that it matters to me a whole lot, but can you provide a "positive vote" to undo some of the nice negative efforts of ErikA thus far to discourage the corrective assistance from taking place in a subject which he has demonstrated that he does not seem qualified to gauge? Thanks! – user48838 Jul 26 '10 at 13:02
  • ErikA, hopefully you are reconsidering your position and will do the "right thing" as this bug is HUGE when it causes folks to have an unstable storage sub-system and OS which counters all the positives of ZFS, if you are unable to reliably access your stored data! – user48838 Jul 26 '10 at 13:05
  • Same for all the blind ZFS zealots who have also offered zero value in pumping up ErikA unqualified comments and denouncing my efforts to bring attention to this HUGE ZFS bug. As the saying goes - "time to man up..." – user48838 Jul 26 '10 at 13:12
  • 1
    @user48838: I would like to vote you positive but: "Vote Up requires 15 reputation" But the server crashed again after writing 228GiB(I think the ARC helped but there is another problem, too?) /var/adm/message tells me: WARNING: ahci0: watchdog port 2 satapkt 0xffffff019dac91d8 timed out Pier from the answer above gave me the advice to update so a newer revision of opensolaris, do you think this would solve this problem or will this casue new problems? – user48937 Jul 26 '10 at 14:16
  • 1
    @user48838 - this has gone far off-topic and as such, this will be my last comment. I have no problem with you bring attention to this bug - my issue (and that of other SF users and mods) was that you blindly copied/pasted nearly identical answers into a whole bunch of ZFS-related questions, while providing zero evidence or reasoning that your stated bug was actually the one the OP was experiencing. So in conclusion, please feel free to draw attention to bugs (ZFS or otherwise) - but please do so in a reasonable and prudent fashion. – EEAA Jul 26 '10 at 15:32
  • ErikA - based on your repeated opinions, caco would not be at the point that they are now in (some/more functionality from when they started) if we went by your negative and continued inaccurate conclusions. If you step back for a reasonable second, I think caco would probably NOT thank you for your efforts thus far in this matter. If you read the other answers carefully in their full context, they are not blind responses - they address ZFS and its production-readiness ("lack of" in light of this very situation rearing its head on a possibly consistent and very repeatable basis). – user48838 Jul 26 '10 at 17:38
  • caco, this is likely a memory (out-of-bounds) situation, so the error message(s) are probably less-than-accurate. I have run into this on OpenSolaris b134 core and possibly at least one or two versions beyond that with the same results/symptoms. Increasing the block size may yield some additional time/functionality short of actually addressing the bug once and for all... – user48838 Jul 26 '10 at 17:49
  • @user48838 - ErikA's comments are valid, please see your personal email - blind reposting regardless of actual question content has not been as helpful as I'm sure you intended. – Chopper3 Jul 26 '10 at 18:13
  • I have not set up my personal email as the collective ill responses from the "moderators" appears to run totally counter to - http://serverfault.com/about (especially against references to "site for system administrators and IT professionals..."). The complete and total inability/unwillingness of the collective "moderators" to review my responses/answers in their full context is a sad statement to the effectiveness of those folks in honoring their own statements of being "professionals..." It makes this seem more like a "click" site, where there is Facebook and other social sites for that... – user48838 Jul 27 '10 at 07:47
  • caco, I did a little quick and dirty regression testing in the opposite direction and it does not seem to look good reading from the ZFS store neither. In the opposite direction, the memory leaking (if it was occurring) did not show up outside of the VM, but the OS did grind to a halt - no reboots for me this time, it just hangs... You might verify this on your end too for the possible anticipation of having to find something other an ZFS... – user48838 Jul 27 '10 at 12:38
  • 1
    Thank you for your time user 48838! My very last hope is that an update to dev will solve my problem, although I guess that nothing will change. Maybe a 0815 Linux with LVM is a better choice for me... – user48937 Jul 27 '10 at 14:46
  • caco, likewise as the ZFS technologies are awesome, if implemented correctly. A lot, if not all, of the pieces are there - they just need to address things like this outstanding bug (especially since it is over 3 years old and its significantly disruptive nature)! – user48838 Jul 27 '10 at 15:07
  • caco, here is yet another bug report that is similar (http://defect.opensolaris.org/bz/show_bug.cgi?id=2366) - maybe even the exact same bug! – user48838 Jul 28 '10 at 21:41
  • Tanks for your advice, but the user wrote following: "Copying large files over CIFS, NFS or iSCSI works fine. No problems no matter how big the file is. "rsync" is also working fine. Just the "cp" seems to make problems." And my server even crashes/crashed on cifs access. But yesterday I made an update to snv_136 and the Server has not crashed for 14 hours(the longest time without a crash since the HDD upgrade). But even if this should have solved my problem, I am not shure about keeping opensolaris because the next HDD upgade will come... – user48937 Jul 29 '10 at 06:55
  • caco, when digging around, it turns out there are a good number of various similar bugs around ARC misbehaving or just not working. I have continued to test various settings as I work with the support folks of a group that have based their product on OpenSolaris (who seems to be having a tough time acknowledging that this is even an issue - instead they keep trying to pin this on everything else). Maybe one more thing to consider... Would you happen to be running an older AMD processor? – user48838 Jul 29 '10 at 13:03
  • You might be right, the user from Bug 2366 had a problem coping large files sounds like ARC... very curious and scary that such a Bug does not get fixed... . I'm not using an older AMD processor the server runs with a Intel(r) Xeon(r) E5502. – user48937 Jul 29 '10 at 15:05
  • caco, it looks like the E5502 is about a year old - not really old, but also not one of the latest either. I have run into OpenSolaris being "picky" with the older AMD models, but the hypervisor may have had a contributing factor too. If it's possible, you might try rehosting your storage subsystem onto a newer CPU platform. That's one of the things on my list to cover, but it will take some efforts for me, so it is not on the very top of my list/schedule. – user48838 Jul 30 '10 at 08:37
  • Wow! It looks like the bugs/leaks are also present in the ports of ZFS! http://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=92&t=6523&start=90 – user48838 Aug 09 '10 at 21:00