3

We've been running ZFS for a couple years now on Solaris 10 on Oracle M5000 Enterprise server with 32 CPU cores and 256GB of memory. We are running a database/application on this server which appears to be read heavy.

We had I/O issues while on UFS and this was resolved by switching to ZFS. We have a NetApp storage unit presenting the disks over fiber channel and then formatted using ZFS at the OS level on a single LUN. At first we had issues with application memory and had to limit the ARC size to 128GB of memory.

Now the issue we started seeing is that the ARC is getting maxed out. During this time the CPU's are overloaded with 0% idle time at times. Application processes grind to a halt and automated processes begin running into each other.

I've been researching the issue for some time and consulted various sources who all seem to believe we just need a bigger machine or to get the vendor to optimize their code. We are looking at purchasing an M10-4 and have been working with the application vendor, but I'm wondering if there might be something else going on.

Any help would be appreciated and let me know if more information is needed.

Below is the output from arc_summary.pl

System Memory:
     Physical RAM:  257614 MB
     Free Memory :  79033 MB
     LotsFree:      4022 MB

ZFS Tunables (/etc/system):
     set zfs:zfs_arc_max = 137438953472

ARC Size:
     Current Size:             131043 MB (arcsize)
     Target Size (Adaptive):   131072 MB (c)
     Min Size (Hard Limit):    64 MB (zfs_arc_min)
     Max Size (Hard Limit):    131072 MB (zfs_arc_max)

ARC Size Breakdown:
     Most Recently Used Cache Size:           6%    8190 MB (p)
     Most Frequently Used Cache Size:        93%    122881 MB (c-p)

ARC Efficency:
     Cache Access Total:             217693264863
     Cache Hit Ratio:      99%       217640359356           [Defined State for buffer]
     Cache Miss Ratio:      0%       52905507       [Undefined State for Buffer]
     REAL Hit Ratio:       67%       146336547931           [MRU/MFU Hits Only]

     Data Demand   Efficiency:    99%
     Data Prefetch Efficiency:    96%

    CACHE HITS BY CACHE LIST:
      Anon:                       32%        71281340234                    [ New Customer, First Cache Hit ]
      Most Recently Used:          0%        172508676 (mru)        [ Return Customer ]
      Most Frequently Used:       67%        146164039255 (mfu)             [ Frequent Customer ]
      Most Recently Used Ghost:    0%        3430197 (mru_ghost)    [ Return Customer Evicted, Now Back ]
      Most Frequently Used Ghost:  0%        19040994 (mfu_ghost)   [ Frequent Customer Evicted, Now Back ]
    CACHE HITS BY DATA TYPE:
      Demand Data:                62%        136382108166
      Prefetch Data:               0%        1145425065
      Demand Metadata:             4%        9980846285
      Prefetch Metadata:          32%        70131979840
    CACHE MISSES BY DATA TYPE:
      Demand Data:                19%        10410690
      Prefetch Data:              72%        38324623
      Demand Metadata:             1%        874219
      Prefetch Metadata:           6%        3295975

Below is output from vmstat:

# vmstat 5
kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr m1 m1 m1 m1   in   sy   cs us sy id
0 0 0 40892888 93939912 256 1411 165 26 26 0 0 8 1 2 0 5651 274471 42913 2 5 93
0 0 0 22113320 81957648 118 1324 3 0 0 0 0 0  0  0  0 5631 313043 58903 2 35 63
0 0 0 22145624 81977312 353 459 0 746 746 0 0 145 0 0 0 5177 379709 58255 2 33 65
0 0 0 22150976 81982088 120 1329 0 0 0 0 0 1  0  0  0 5200 314711 59490 2 33 65
0 0 0 22147600 81981680 504 1834 0 8 5 0 0 5  0  0  0 5197 334201 58339 2 32 66
0 0 0 22146160 81982544 715 610 0 5 3 0 0  2  0  0  0 6296 364301 58900 2 35 63
0 0 0 22116584 81960496 678 1928 0 3 3 0 0 1  0  0  0 5178 351160 58947 2 34 64
1 0 0 22107080 81949528 95 1095 0 0 0 0 0  0  0  0  0 4531 309206 58450 2 35 63

Below is output from zpool iostat:

# zpool iostat ntapdatatel 5
            capacity     operations    bandwidth
pool         alloc   free   read  write   read  write
-----------  -----  -----  -----  -----  -----  -----
ntapdatatel  1.07T   437G     10     13  1.07M  1.01M
ntapdatatel  1.07T   437G      0      0  51.2K  4.00K
ntapdatatel  1.07T   437G      0      0      0      0
ntapdatatel  1.07T   437G      0     95      0  6.47M
ntapdatatel  1.07T   437G      0      0      0      0
ntapdatatel  1.07T   437G      0      0      0      0
ntapdatatel  1.07T   437G      0      0      0      0
ntapdatatel  1.07T   437G      0      0      0      0
ntapdatatel  1.07T   437G      0      0  25.6K      0
ntapdatatel  1.07T   437G      0     87      0  5.16M

and the file system properties:

NAME         PROPERTY              VALUE                  SOURCE
ntapdatatel  type                  filesystem             -
ntapdatatel  creation              Sun Oct 28 17:09 2012  -
ntapdatatel  used                  1.07T                  -
ntapdatatel  available             413G                   -
ntapdatatel  referenced            432G                   -
ntapdatatel  compressratio         1.00x                  -
ntapdatatel  mounted               yes                    -
ntapdatatel  quota                 none                   default
ntapdatatel  reservation           none                   default
ntapdatatel  recordsize            128K                   default
ntapdatatel  mountpoint            /ntapdatatel           local
ntapdatatel  sharenfs              off                    default
ntapdatatel  checksum              on                     default
ntapdatatel  compression           off                    default
ntapdatatel  atime                 on                     default
ntapdatatel  devices               on                     default
ntapdatatel  exec                  on                     default
ntapdatatel  setuid                on                     default
ntapdatatel  readonly              off                    default
ntapdatatel  zoned                 off                    default
ntapdatatel  snapdir               hidden                 default
ntapdatatel  aclmode               discard                default
ntapdatatel  aclinherit            restricted             default
ntapdatatel  canmount              on                     default
ntapdatatel  shareiscsi            off                    default
ntapdatatel  xattr                 on                     default
ntapdatatel  copies                1                      default
ntapdatatel  version               5                      -
ntapdatatel  utf8only              off                    -
ntapdatatel  normalization         none                   -
ntapdatatel  casesensitivity       sensitive              -
ntapdatatel  vscan                 off                    default
ntapdatatel  nbmand                off                    default
ntapdatatel  sharesmb              off                    default
ntapdatatel  refquota              none                   default
ntapdatatel  refreservation        none                   default
ntapdatatel  primarycache          all                    default
ntapdatatel  secondarycache        all                    default
ntapdatatel  usedbysnapshots       0                      -
ntapdatatel  usedbydataset         432G                   -
ntapdatatel  usedbychildren        666G                   -
ntapdatatel  usedbyrefreservation  0                      -
ntapdatatel  logbias               latency                default
ntapdatatel  sync                  standard               default
ntapdatatel  rekeydate             -                      default
ntapdatatel  rstchown              on                     default
emaN_yalpsiD
  • 53
  • 5

2 Answers2

3

It may not be zfs - you have lots of free memory, so consider this possibility -

echo 'pg_contig_disable/D' | mdb -k

If the output is:

echo pg_contig_disable/D | mdb -k
pg_contig_disable:
pg_contig_disable:              0

You may be experiencing a sort of NUMA issue. Solaris 10 tries to facilitate faster memory access by setting up blocks of memory for cache efficiency. This, when you have lots of memory and oracle, results in a wierd situation - just like you describe. After a month or so of uptime, not much cpu user usage, lots of system cpu time, and the system grinds to a halt. Long term, set the kernel parameter pg_contig_disable to 1.

The short term fix is to reboot. If reboot helps you need to set the kernel parm. This is a known Solaris 10 issue on systems with lots of free memory.

jim mcnamara
  • 429
  • 3
  • 8
  • Checking into this now. That parameter is off on our system so it's certainly a possibility. I'm waiting for another hit on the system to check for high page_trylocks. Looks like that's a symptom of this particular issue. – emaN_yalpsiD Oct 28 '14 at 14:51
  • FYI, we are on a UniData database and not Oracle, but guess we'll see if the issue expands beyond Oracle DB's. – emaN_yalpsiD Oct 28 '14 at 15:14
2

Thanks to jim mcnamara for pointing me in the right direction. I wasn't seeing symptoms consistent with the pg_contig_disable issue, but it did lead me to an issue with zfetch.

I found the same issue we were having on the following site: http://solaristalk.blogspot.com/2014_05_01_archive.html

This led to a tuning article on the Oracle site describing why ZFS prefetching was an issue for us: http://docs.oracle.com/cd/E26502_01/html/E29022/chapterzfs-4.html

We were seeing dmu_zfetch_find at the top of the mutex list using lockstat during heavy load. I've since disabled prefetching in our ZFS implementation. I'll be rebooting this evening to ensure that everything gets cleared out and we start fresh.

Hopefully this is the ticket. We may do some testing with pg_contig_disable later on if we are still having issues just in case it helps.

emaN_yalpsiD
  • 53
  • 5