Mailing List Archive: Varnish: Misc

varnish and ram size

Index | Next | Previous | View Threaded

shahab371 at gmail

Apr 28, 2012, 4:28 AM

Post #1 of 6 (469 views)
Permalink

varnish and ram size

Hi

I have configured varnish to use 1GB of cache:
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G

but it seems that varnish aggresively uses the whole 4Gb Ram of the machine
and then crashes, followed by a restart.
here is the log

Apr 28 13:18:47 varnish varnishd[2952]: Child (5852) not responding to
ping, killing it.
Apr 28 13:18:49 varnish varnishd[2952]: Child (5852) not responding to
ping, killing it.
Apr 28 13:18:49 varnish varnishd[2952]: Child (5852) died signal=3
Apr 28 13:18:49 varnish varnishd[2952]: Child cleanup complete
Apr 28 13:18:49 varnish varnishd[2952]: child (9522) Started
Apr 28 13:18:49 varnish varnishd[2952]: Child (9522) said
Apr 28 13:18:49 varnish varnishd[2952]: Child (9522) said Child starts
Apr 28 13:18:49 varnish varnishd[2952]: Child (9522) said managed to mmap
6275375104 bytes of 6275375104

what am I doing wrong? is there a way prohibit that or at least tell
varnish not to use the whole cache?

Thank you in advance

Shahab B.

tfheen at varnish-software

May 2, 2012, 3:31 AM

Post #2 of 6 (446 views)
Permalink

Re: varnish and ram size [In reply to]

]] shahab bakhtiyari

Hi,

> what am I doing wrong? is there a way prohibit that or at least tell
> varnish not to use the whole cache?

Please provide the output of varnishstat -1 when it's using more than
the allocated amount of resources and we might be able to figure out
what's wrong.

--
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

shahab371 at gmail

May 2, 2012, 7:37 AM

Post #3 of 6 (430 views)
Permalink

Re: varnish and ram size [In reply to]

On 2 May 2012 15:54, shahab bakhtiyari <shahab371 [at] gmail> wrote:

> Hi
>
> here you can see outputs from top, and varnishstat -1
>
> top:
> 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd
>
>
>
>
>
> root [at] varnis:~# varnishstat -1
> client_conn 259312 158.89 Client connections accepted
> client_drop 0 0.00 Connection dropped, no sess/wrk
> client_req 259212 158.83 Client requests received
> cache_hit 33921 20.78 Cache hits
> cache_hitpass 0 0.00 Cache hits for pass
> cache_miss 225291 138.05 Cache misses
> backend_conn 43774 26.82 Backend conn. success
> backend_unhealthy 0 0.00 Backend conn. not attempted
> backend_busy 0 0.00 Backend conn. too many
> backend_fail 0 0.00 Backend conn. failures
> backend_reuse 181617 111.28 Backend conn. reuses
> backend_toolate 0 0.00 Backend conn. was closed
> backend_recycle 181617 111.28 Backend conn. recycles
> backend_unused 0 0.00 Backend conn. unused
> fetch_head 0 0.00 Fetch head
> fetch_length 225291 138.05 Fetch with Length
> fetch_chunked 0 0.00 Fetch chunked
> fetch_eof 0 0.00 Fetch EOF
> fetch_bad 0 0.00 Fetch had bad headers
> fetch_close 0 0.00 Fetch wanted close
> fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
> fetch_zero 0 0.00 Fetch zero len
> fetch_failed 0 0.00 Fetch failed
> n_sess_mem 351 . N struct sess_mem
> n_sess 102 . N struct sess
> n_object 220554 . N struct object
> n_vampireobject 0 . N unresurrected objects
> n_objectcore 220557 . N struct objectcore
> n_objecthead 216397 . N struct objecthead
> n_smf 441172 . N struct smf
> n_smf_frag 0 . N small free smf
> n_smf_large 2 . N large free smf
> n_vbe_conn 100 . N struct vbe_conn
> n_wrk 121 . N worker threads
> n_wrk_create 169 0.10 N worker threads created
> n_wrk_failed 0 0.00 N worker threads not created
> n_wrk_max 951160 582.82 N worker threads limited
> n_wrk_queue 0 0.00 N queued work requests
> n_wrk_overflow 2082 1.28 N overflowed work requests
> n_wrk_drop 0 0.00 N dropped work requests
> n_backend 1 . N backends
> n_expired 4737 . N expired objects
> n_lru_nuked 0 . N LRU nuked objects
> n_lru_saved 0 . N LRU saved objects
> n_lru_moved 26276 . N LRU moved objects
> n_deathrow 0 . N objects on deathrow
> losthdr 0 0.00 HTTP header overflows
> n_objsendfile 0 0.00 Objects sent with sendfile
> n_objwrite 259207 158.83 Objects sent with write
> n_objoverflow 0 0.00 Objects overflowing workspace
> s_sess 259212 158.83 Total Sessions
> s_req 259212 158.83 Total Requests
> s_pipe 0 0.00 Total pipe
> s_pass 0 0.00 Total pass
> s_fetch 225291 138.05 Total fetch
> s_hdrbytes 119525246 73238.51 Total header bytes
> s_bodybytes 2748370979 1684050.84 Total body bytes
> sess_closed 259212 158.83 Session Closed
> sess_pipeline 0 0.00 Session Pipeline
> sess_readahead 0 0.00 Session Read Ahead
> sess_linger 0 0.00 Session Linger
> sess_herd 0 0.00 Session herd
> shm_records 21397311 13111.10 SHM records
> shm_writes 1445766 885.89 SHM writes
> shm_flushes 27 0.02 SHM flushes due to overflow
> shm_cont 9812 6.01 SHM MTX contention
> shm_cycles 9 0.01 SHM cycles through buffer
> sm_nreq 450644 276.13 allocator requests
> sm_nobj 441170 . outstanding allocations
> sm_balloc 3777261568 . bytes allocated
> sm_bfree 743198720 . bytes free
> sma_nreq 0 0.00 SMA allocator requests
> sma_nobj 0 . SMA outstanding allocations
> sma_nbytes 0 . SMA outstanding bytes
> sma_balloc 0 . SMA bytes allocated
> sma_bfree 0 . SMA bytes free
> sms_nreq 0 0.00 SMS allocator requests
> sms_nobj 0 . SMS outstanding allocations
> sms_nbytes 0 . SMS outstanding bytes
> sms_balloc 0 . SMS bytes allocated
> sms_bfree 0 . SMS bytes freed
> backend_req 225390 138.11 Backend requests made
> n_vcl 1 0.00 N vcl total
> n_vcl_avail 1 0.00 N vcl available
> n_vcl_discard 0 0.00 N vcl discarded
> n_purge 1 . N total active purges
> n_purge_add 2 0.00 N new purges added
> n_purge_retire 1 0.00 N old purges deleted
> n_purge_obj_test 0 0.00 N objects tested
> n_purge_re_test 0 0.00 N regexps tested against
> n_purge_dups 0 0.00 N duplicate purges removed
> hcb_nolock 259312 158.89 HCB Lookups without lock
> hcb_lock 217430 133.23 HCB Lookups with lock
> hcb_insert 217428 133.23 HCB Inserts
> esi_parse 0 0.00 Objects ESI parsed (unlock)
> esi_errors 0 0.00 ESI parse errors (unlock)
> accept_fail 0 0.00 Accept failures
> client_drop_late 0 0.00 Connection dropped late
> uptime 1632 1.00 Client uptime
>
>
>
>
> On 2 May 2012 12:31, Tollef Fog Heen <tfheen [at] varnish-software> wrote:
>
>> ]] shahab bakhtiyari
>>
>> Hi,
>>
>> > what am I doing wrong? is there a way prohibit that or at least tell
>> > varnish not to use the whole cache?
>>
>> Please provide the output of varnishstat -1 when it's using more than
>> the allocated amount of resources and we might be able to figure out
>> what's wrong.
>>
>> --
>> Tollef Fog Heen
>> Technical lead, Varnish Software
>> t: +47 21 98 92 64
>>
>
>

shahab371 at gmail

May 2, 2012, 7:45 AM

Post #4 of 6 (442 views)
Permalink

Re: varnish and ram size [In reply to]

Hi

here you can see outputs from top, and varnishstat -1

top:
12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd

root [at] varnis:~# varnishstat -1
client_conn 259312 158.89 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 259212 158.83 Client requests received
cache_hit 33921 20.78 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 225291 138.05 Cache misses
backend_conn 43774 26.82 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 0 0.00 Backend conn. failures
backend_reuse 181617 111.28 Backend conn. reuses
backend_toolate 0 0.00 Backend conn. was closed
backend_recycle 181617 111.28 Backend conn. recycles
backend_unused 0 0.00 Backend conn. unused
fetch_head 0 0.00 Fetch head
fetch_length 225291 138.05 Fetch with Length
fetch_chunked 0 0.00 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 0 0.00 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 0 0.00 Fetch failed
n_sess_mem 351 . N struct sess_mem
n_sess 102 . N struct sess
n_object 220554 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 220557 . N struct objectcore
n_objecthead 216397 . N struct objecthead
n_smf 441172 . N struct smf
n_smf_frag 0 . N small free smf
n_smf_large 2 . N large free smf
n_vbe_conn 100 . N struct vbe_conn
n_wrk 121 . N worker threads
n_wrk_create 169 0.10 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 951160 582.82 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 2082 1.28 N overflowed work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 1 . N backends
n_expired 4737 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_lru_moved 26276 . N LRU moved objects
n_deathrow 0 . N objects on deathrow
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 259207 158.83 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 259212 158.83 Total Sessions
s_req 259212 158.83 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 225291 138.05 Total fetch
s_hdrbytes 119525246 73238.51 Total header bytes
s_bodybytes 2748370979 1684050.84 Total body bytes
sess_closed 259212 158.83 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 0 0.00 Session Linger
sess_herd 0 0.00 Session herd
shm_records 21397311 13111.10 SHM records
shm_writes 1445766 885.89 SHM writes
shm_flushes 27 0.02 SHM flushes due to overflow
shm_cont 9812 6.01 SHM MTX contention
shm_cycles 9 0.01 SHM cycles through buffer
sm_nreq 450644 276.13 allocator requests
sm_nobj 441170 . outstanding allocations
sm_balloc 3777261568 . bytes allocated
sm_bfree 743198720 . bytes free
sma_nreq 0 0.00 SMA allocator requests
sma_nobj 0 . SMA outstanding allocations
sma_nbytes 0 . SMA outstanding bytes
sma_balloc 0 . SMA bytes allocated
sma_bfree 0 . SMA bytes free
sms_nreq 0 0.00 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 0 . SMS bytes allocated
sms_bfree 0 . SMS bytes freed
backend_req 225390 138.11 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_purge 1 . N total active purges
n_purge_add 2 0.00 N new purges added
n_purge_retire 1 0.00 N old purges deleted
n_purge_obj_test 0 0.00 N objects tested
n_purge_re_test 0 0.00 N regexps tested against
n_purge_dups 0 0.00 N duplicate purges removed
hcb_nolock 259312 158.89 HCB Lookups without lock
hcb_lock 217430 133.23 HCB Lookups with lock
hcb_insert 217428 133.23 HCB Inserts
esi_parse 0 0.00 Objects ESI parsed (unlock)
esi_errors 0 0.00 ESI parse errors (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 1632 1.00 Client uptime

On 2 May 2012 12:31, Tollef Fog Heen <tfheen [at] varnish-software> wrote:

> ]] shahab bakhtiyari
>
> Hi,
>
> > what am I doing wrong? is there a way prohibit that or at least tell
> > varnish not to use the whole cache?
>
> Please provide the output of varnishstat -1 when it's using more than
> the allocated amount of resources and we might be able to figure out
> what's wrong.
>
> --
> Tollef Fog Heen
> Technical lead, Varnish Software
> t: +47 21 98 92 64
>

tfheen at varnish-software

May 4, 2012, 1:32 AM

Post #5 of 6 (423 views)
Permalink

Re: varnish and ram size [In reply to]

]] shahab bakhtiyari

> here you can see outputs from top, and varnishstat -1
>
> top:
> 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd

> sm_balloc 3777261568 . bytes allocated
> sm_bfree 743198720 . bytes free

These numbers look consistent. What -s argument are you giving
varnishd, and does the storage file already exist on disk with a
different size? If so, removing it and letting Varnish recreate it
might be helpful.

Cheers,
--
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

shahab371 at gmail

May 4, 2012, 1:58 AM

Post #6 of 6 (427 views)
Permalink

Re: varnish and ram size [In reply to]

On 4 May 2012 10:32, Tollef Fog Heen <tfheen [at] varnish-software> wrote:

> ]] shahab bakhtiyari
>
> > here you can see outputs from top, and varnishstat -1
> >
> > top:
> > 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd
>
> > sm_balloc 3777261568 . bytes allocated
> > sm_bfree 743198720 . bytes free
>
> These numbers look consistent. What -s argument are you giving
> varnishd,

-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

> and does the storage file already exist on disk with a
> different size?

no it does not exist. I upgraded to version 3.0.2 yesterday. now I notice
that when I start varnish, it creates a vcl.somechars.so file instead of
varnish_storage.bin . Does it have something to say?

If so, removing it and letting Varnish recreate it
> might be helpful.
>
>
generally, isn't there any way to tell varnish how much ram it should use?
something like malloc option, but malloc is well, ment to be used when
varnish is supposed to use ONLY ram, right?

> Cheers,
> --
> Tollef Fog Heen
> Technical lead, Varnish Software
> t: +47 21 98 92 64
>

thanks in advance
Shahab

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads