We are using varnish 3.0.3-3.5 on amzn ec2 and it is allocated around 3G of malloc. It starts with normal caching of object but when it hits a certain no of object+objecthead(around 14k), n_lru_nuked objects starts to increase rapidly and no of objects becomes saturated. Tried also with version 3.0.4-1 but didn't work. Please help me find the problem. Let me know if any other information is required.
Following is the setting:
DAEMON_OPTS="-a 0.0.0.0:80,:8980 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-s malloc,3G"
Stats info:
client_conn 1119377 4.89 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 7253764 31.69 Client requests received
cache_hit 276698 1.21 Cache hits
cache_hitpass 205516 0.90 Cache hits for pass
cache_miss 2478864 10.83 Cache misses
backend_conn 405876 1.77 Backend conn. success
backend_unhealthy 3421 0.01 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 3938 0.02 Backend conn. failures
backend_reuse 6528731 28.53 Backend conn. reuses
backend_toolate 50598 0.22 Backend conn. was closed
backend_recycle 6579393 28.75 Backend conn. recycles
backend_retry 1351 0.01 Backend conn. retry
fetch_head 111720 0.49 Fetch head
fetch_length 216430 0.95 Fetch with Length
fetch_chunked 6486185 28.34 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 74820 0.33 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 17 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 9986 0.04 Fetch no body (304)
n_sess_mem 107 . N struct sess_mem
n_sess 37 . N struct sess
n_object 4373 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 4384 . N struct objectcore
n_objecthead 8041 . N struct objecthead
n_waitinglist 3180 . N struct waitinglist
n_vbc 75 . N struct vbc
n_wrk 41 . N worker threads
n_wrk_create 1585 0.01 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 12335 0.05 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 34 . N backends
n_expired 501853 . N expired objects
n_lru_nuked 1972544 . N LRU nuked objects
n_lru_moved 249223 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 7072900 30.90 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 1119369 4.89 Total Sessions
s_req 7253764 31.69 Total Requests
s_pipe 39412 0.17 Total pipe
s_pass 4421895 19.32 Total pass
s_fetch 6899124 30.14 Total fetch
s_hdrbytes 3244163451 14174.64 Total header bytes
s_bodybytes 54294899374 237229.27 Total body bytes
sess_closed 275785 1.20 Session Closed
sess_pipeline 2 0.00 Session Pipeline
sess_readahead 3 0.00 Session Read Ahead
sess_linger 7181910 31.38 Session Linger
sess_herd 3441115 15.04 Session herd
shm_records 662367830 2894.07 SHM records
shm_writes 31444934 137.39 SHM writes
shm_flushes 1 0.00 SHM flushes due to overflow
shm_cont 5840 0.03 SHM MTX contention
shm_cycles 325 0.00 SHM cycles through buffer
sms_nreq 32172 0.14 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 12256989 . SMS bytes allocated
sms_bfree 12256989 . SMS bytes freed
backend_req 6900427 30.15 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 2961085 12.94 HCB Lookups without lock
hcb_lock 2464684 10.77 HCB Lookups with lock
hcb_insert 2464679 10.77 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 228871 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 5852234 25.57 Gunzip operations
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 96516 0.42 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 1 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 21844471 95.44 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 1 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 14105325 61.63 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 4926255 21.52 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 6438 0.03 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 1119443 4.89 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 1136901 4.97 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 499424 2.18 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 12215 0.05 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 2 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 7893519 34.49 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 2465018 10.77 Created locks
LCK.objhdr.destroy 2457038 10.74 Destroyed locks
LCK.objhdr.locks 16808877 73.44 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 5182011 22.64 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 4451317 19.45 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 76262 0.33 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 5182052 22.64 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 3155814 13.79 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 819553 3.58 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 34 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 14391476 62.88 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMF.s0.c_req 6836951 29.87 Allocator requests
SMF.s0.c_fail 1972544 8.62 Allocator failures
SMF.s0.c_bytes 327625674752 1431486.19 Bytes allocated
SMF.s0.c_freed 327564849152 1431220.42 Bytes freed
SMF.s0.g_alloc 8403 . Allocations outstanding
SMF.s0.g_bytes 60825600 . Bytes outstanding
SMF.s0.g_space 44032000 . Bytes available
SMF.s0.g_smf 8805 . N struct smf
SMF.s0.g_smf_frag 402 . N small free smf
SMF.s0.g_smf_large 0 . N large free smf
SMA.Transient.c_req 8848148 38.66 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 564174348191 2465032.04 Bytes allocated
SMA.Transient.c_freed 564174332082 2465031.97 Bytes freed
SMA.Transient.g_alloc 10 . Allocations outstanding
SMA.Transient.g_bytes 16109 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
