Mailing List Archive: Varnish: Misc

Varnish memory usage

Index | Next | Previous | View Threaded

michaell at dazzit

Jun 9, 2010, 5:24 PM

Post #1 of 16 (6394 views)
Permalink

Varnish memory usage

I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely
fill Varnish's cache.

"top" shows:

1486m

769 nobody 20 0 1486m 1.3g 81m S 0.0 78.6 0:03.71 varnishd

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

michaell at dazzit

Jun 9, 2010, 5:29 PM

Post #2 of 16 (6352 views)
Permalink

Re: Varnish memory usage [In reply to]

What a way to start! Obviously, I accidentally hit "send." Let's try
this again...

I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely
fill Varnish's cache. We've defined that cache as "malloc,1G". After
running our test, "top" shows:

1486m VIRT memory
1.3g RES memory
81m SHR memory

Either I've mis-configured something or mis-understood something.
Either way, help is appreciated.

In case it's of use the output of varnishstat follows.

0+05:22:49 domU-12-31-38-00-9D-27
Hitrate ratio: 10 100 1000
Hitrate avg: 0.0381 0.0813 0.0705

3889 0.00 0.20 Client connections accepted
227735 0.99 11.76 Client requests received
10916 0.00 0.56 Cache hits
215592 0.00 11.13 Cache misses
211160 0.00 10.90 Backend conn. success
12 0.00 0.00 Backend conn. failures
5646 0.99 0.29 Backend conn. reuses
211144 0.00 10.90 Backend conn. was closed
216791 0.99 11.19 Backend conn. recycles
4575 0.99 0.24 Fetch with Length
212231 0.00 10.96 Fetch chunked
31 . . N struct sess_mem
2 . . N struct sess
205011 . . N struct object
205014 . . N struct objectcore
205014 . . N struct objecthead
1 . . N struct vbe_conn
500 . . N worker threads
500 0.00 0.03 N worker threads created
1259710 0.00 65.04 N worker threads limited
2 . . N backends
10569 . . N expired objects
4425 . . N LRU moved objects
216589 0.99 11.18 Objects sent with write
3889 0.00 0.20 Total Sessions
227735 0.99 11.76 Total Requests
1226 0.99 0.06 Total pass
216806 0.99 11.19 Total fetch
97606086 221.79 5039.29 Total header bytes
833650072 2.97 43040.43 Total body bytes
13 0.00 0.00 Session Closed
231611 0.99 11.96 Session herd
21199614 59.41 1094.51 SHM records
697327 2.97 36.00 SHM writes
38 0.00 0.00 SHM MTX contention
8 0.00 0.00 SHM cycles through buffer
432162 0.99 22.31 SMA allocator requests
410022 . . SMA outstanding allocations
966403669 . . SMA outstanding bytes
27988956353 . . SMA bytes allocated
27022552684 . . SMA bytes free
13 0.00 0.00 SMS allocator requests
3670 . . SMS bytes allocated
3670 . . SMS bytes freed
216806 0.99 11.19 Backend requests made
1 0.00 0.00 N vcl total
1 0.00 0.00 N vcl available
1 . . N total active purges
1 0.00 0.00 N new purges added
19369 1.98 1.00 Client uptime

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

kb+varnish at slide

Jun 9, 2010, 6:31 PM

Post #3 of 16 (6344 views)
Permalink

Re: Varnish memory usage [In reply to]

> 966403669 . . SMA outstanding bytes

There's your 1GB cache buffer, which is what you're setting. Obviously, Varnish needs memory to do other things, so you'll see a fixed couple of hundred extra megs. Also, each thread (e.g., worker) will allocate its own memory, including a stack. This memory usage will scale and contract as load spawns and reaps threads. This is true of any server, especially examples such as Squid and MySQLd.

Linux (including 10.04) sets a default stack size of 8MB:

-s: stack size (kbytes) 8192

With 500 threads, you could in theory see up to 4GB of VIRT.

Run "pmap -x PID" on the child process (or grandchildren), and you will see where the memory goes. It should reflect the above -- you'll see 500+ 8MB stacks, plus a lot of other 1MB allocations, which is the block size used by jemalloc IIRC.

If you really only want to use 1G, specify something like "malloc,800M".

Hope it helps,
--
Ken

On Jun 9, 2010, at 5:29 PM, Michael Lenaghan wrote:

> What a way to start! Obviously, I accidentally hit "send." Let's try
> this again...
>
> I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely
> fill Varnish's cache. We've defined that cache as "malloc,1G". After
> running our test, "top" shows:
>
> 1486m VIRT memory
> 1.3g RES memory
> 81m SHR memory
>
> Either I've mis-configured something or mis-understood something.
> Either way, help is appreciated.
>
> In case it's of use the output of varnishstat follows.
>
> 0+05:22:49 domU-12-31-38-00-9D-27
> Hitrate ratio: 10 100 1000
> Hitrate avg: 0.0381 0.0813 0.0705
>
> 3889 0.00 0.20 Client connections accepted
> 227735 0.99 11.76 Client requests received
> 10916 0.00 0.56 Cache hits
> 215592 0.00 11.13 Cache misses
> 211160 0.00 10.90 Backend conn. success
> 12 0.00 0.00 Backend conn. failures
> 5646 0.99 0.29 Backend conn. reuses
> 211144 0.00 10.90 Backend conn. was closed
> 216791 0.99 11.19 Backend conn. recycles
> 4575 0.99 0.24 Fetch with Length
> 212231 0.00 10.96 Fetch chunked
> 31 . . N struct sess_mem
> 2 . . N struct sess
> 205011 . . N struct object
> 205014 . . N struct objectcore
> 205014 . . N struct objecthead
> 1 . . N struct vbe_conn
> 500 . . N worker threads
> 500 0.00 0.03 N worker threads created
> 1259710 0.00 65.04 N worker threads limited
> 2 . . N backends
> 10569 . . N expired objects
> 4425 . . N LRU moved objects
> 216589 0.99 11.18 Objects sent with write
> 3889 0.00 0.20 Total Sessions
> 227735 0.99 11.76 Total Requests
> 1226 0.99 0.06 Total pass
> 216806 0.99 11.19 Total fetch
> 97606086 221.79 5039.29 Total header bytes
> 833650072 2.97 43040.43 Total body bytes
> 13 0.00 0.00 Session Closed
> 231611 0.99 11.96 Session herd
> 21199614 59.41 1094.51 SHM records
> 697327 2.97 36.00 SHM writes
> 38 0.00 0.00 SHM MTX contention
> 8 0.00 0.00 SHM cycles through buffer
> 432162 0.99 22.31 SMA allocator requests
> 410022 . . SMA outstanding allocations
> 966403669 . . SMA outstanding bytes
> 27988956353 . . SMA bytes allocated
> 27022552684 . . SMA bytes free
> 13 0.00 0.00 SMS allocator requests
> 3670 . . SMS bytes allocated
> 3670 . . SMS bytes freed
> 216806 0.99 11.19 Backend requests made
> 1 0.00 0.00 N vcl total
> 1 0.00 0.00 N vcl available
> 1 . . N total active purges
> 1 0.00 0.00 N new purges added
> 19369 1.98 1.00 Client uptime
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc [at] varnish-cache
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

michaell at dazzit

Jun 9, 2010, 7:16 PM

Post #4 of 16 (6359 views)
Permalink

Re: Varnish memory usage [In reply to]

Thanks for the reply, Ken.

>> � 966403669 � � � � �. � � � � � �. � SMA outstanding bytes
>
> There's your 1GB cache buffer, which is what you're setting.

OK, I see that "SMA bytes allocated" - "SMA bytes free" = "SMA
outstanding bytes". Perhaps, in the process of allocating and freeing,
memory has become fragmented--leaving unusable holes...?

> Linux (including 10.04) sets a default stack size of 8MB:
>
> -s: stack size (kbytes) � � � �8192
>
> With 500 threads, you could in theory see up to 4GB of VIRT.

Varnish sets the stack size to 65K on this system by default:

thread_pool_stack 65536 [bytes]
Default is -1
Worker thread stack size.

That works out to 32 MB for 500 threads. And I force the creation of
all threads up front in order to avoid surprise memory allocations
under load.

> Run "pmap -x PID" on the child process (or grandchildren), and you will see where the memory goes.
> It should reflect the above -- you'll see 500+ 8MB stacks, plus a lot of other 1MB allocations, which
> is the block size used by jemalloc IIRC.

Using pmap I can confirm the 65 KB thread stacks. After pushing
Varnish a little further--it's at 1.6 GB virtual memory now--I also
see one line like this:

52f00000 1432576 - - - rw--- [ anon ]

That's a single 1.4 GB allocation?

The next biggest allocations (in address order) are:

ac3f6000 10240 - - - rw--- [ anon ]
acdf7000 10240 - - - rw--- [ anon ]
ad7f8000 10240 - - - rw--- [ anon ]
ae200000 12288 - - - rw--- [ anon ]
afaf0000 10304 - - - rw--- [ anon ]
b05ed000 10240 - - - rw--- [ anon ]
b0fee000 10240 - - - rw--- [ anon ]
b19ef000 10240 - - - rw--- [ anon ]
b23ef000 81988 - - - rw-s- _.vsl

Those ten allocations account for nearly all of the memory used by
Varnish. (The first line alone is just under 90%.)

> If you really only want to use 1G, specify something like "malloc,800M".

Yes, I can try that. And in production I think I can set things up so
that I guarantee Varnish never fills its cache. But I wanted to see
how Varnish performed at the limit. And I'd like to get as close to
the limit as I safely can.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

michaell at dazzit

Jun 10, 2010, 9:03 AM

Post #5 of 16 (6398 views)
Permalink

Re: Varnish memory usage [In reply to]

Hello, all. Just thought I'd share our results:

* With "malloc,512M" Varnish seemed to stabilize at ~935 MB virtual memory.

* With "malloc,850M" Varnish seemed to stabilize at ~1,330 MB virtual memory.

* With "malloc,1G" Varnish seemed to stabilize at ~1,630 MB virtual memory.

No doubt all of this is sensitive to our test patterns, etc. Still,
might be useful.

Thanks again, Ken.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

richard.chiswell at mangahigh

Jun 10, 2010, 10:20 AM

Post #6 of 16 (6372 views)
Permalink

Re: Varnish memory usage [In reply to]

Hi Ken,

That's useful - how many threads and thread pools are you using?

Rich

On 10/06/2010 17:03, Michael Lenaghan wrote:
> Hello, all. Just thought I'd share our results:
>
> * With "malloc,512M" Varnish seemed to stabilize at ~935 MB virtual memory.
>
> * With "malloc,850M" Varnish seemed to stabilize at ~1,330 MB virtual memory.
>
> * With "malloc,1G" Varnish seemed to stabilize at ~1,630 MB virtual memory.
>
> No doubt all of this is sensitive to our test patterns, etc. Still,
> might be useful.
>
> Thanks again, Ken.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc [at] varnish-cache
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

michaell at dazzit

Jun 10, 2010, 11:08 AM

Post #7 of 16 (6350 views)
Permalink

Re: Varnish memory usage [In reply to]

> That's useful - how many threads and thread pools are you using?

Hello, Rich. I think the question was actually aimed at me...

I'm using:

-h classic,333333
-p listen_depth=2048
-p thread_pools=1
-p thread_pool_min=500
-p thread_pool_max=500
-p sess_timeout=600
-p session_linger=0

(The hash table is bigger than I need at this point, but that's what
the test used.)

In other news: I ran the test on an Amazon m1.small instance type
("equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon
processor" with 1.7 GB of memory). In a separate performance test that
box could achieve ~6,000 RPS* on a tiny document (~128 B), or ~10,000
RPS if requests were pipelined. Large documents (> ~32 KB) could
saturate the network pipe (~80,000 KB/s) at a significantly lower RPS.
CPU utilization never exceeded ~38%, though there's a significant
discrepancy between the numbers reported by top and the numbers
reported by Amazon's CloudWatch.

===

*RPS means "requests per second," not "sessions per second." For what
we're doing I wasn't interested in measuring TCP connection overhead.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

jan at architechs

Jan 18, 2012, 8:09 AM

Post #8 of 16 (4324 views)
Permalink

Re: Varnish memory usage [In reply to]

Hi,

On Wed, Jan 18, 2012 at 16:05, Andreas Plesner Jacobsen <apj [at] mutt> wrote:
> We'd like to see varnishstat -1 and your vcl too.

proxy03 varnish # varnishstat -1
client_conn 11139628 42.95 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 48047460 185.27 Client requests received
cache_hit 42313341 163.16 Cache hits
cache_hitpass 59583 0.23 Cache hits for pass
cache_miss 4988042 19.23 Cache misses
backend_conn 3294448 12.70 Backend conn. success
backend_unhealthy 36426 0.14 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 211 0.00 Backend conn. failures
backend_reuse 2402445 9.26 Backend conn. reuses
backend_toolate 2748 0.01 Backend conn. was closed
backend_recycle 2405196 9.27 Backend conn. recycles
backend_retry 144 0.00 Backend conn. retry
fetch_head 478 0.00 Fetch head
fetch_length 5138161 19.81 Fetch with Length
fetch_chunked 475194 1.83 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 2985 0.01 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 45 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 329 0.00 Fetch no body (304)
n_sess_mem 2580 . N struct sess_mem
n_sess 1315 . N struct sess
n_object 761556 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 761679 . N struct objectcore
n_objecthead 523573 . N struct objecthead
n_waitinglist 4299 . N struct waitinglist
n_vbc 12 . N struct vbc
n_wrk 136 . N worker threads
n_wrk_create 1520 0.01 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 26664 0.10 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 19 . N backends
n_expired 902775 . N expired objects
n_lru_nuked 3222521 . N LRU nuked objects
n_lru_moved 395588 . N LRU moved objects
losthdr 37 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 40679068 156.86 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 11139613 42.95 Total Sessions
s_req 48047460 185.27 Total Requests
s_pipe 204 0.00 Total pipe
s_pass 747967 2.88 Total pass
s_fetch 5617078 21.66 Total fetch
s_hdrbytes 18002004233 69414.68 Total header bytes
s_bodybytes 496206310294 1913342.76 Total body bytes
sess_closed 1772616 6.84 Session Closed
sess_pipeline 198838 0.77 Session Pipeline
sess_readahead 48712 0.19 Session Read Ahead
sess_linger 47017485 181.30 Session Linger
sess_herd 42381685 163.42 Session herd
shm_records 2342457931 9032.38 SHM records
shm_writes 138479662 533.97 SHM writes
shm_flushes 14 0.00 SHM flushes due to overflow
shm_cont 3181498 12.27 SHM MTX contention
shm_cycles 920 0.00 SHM cycles through buffer
sms_nreq 89029 0.34 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 30624944 . SMS bytes allocated
sms_bfree 30624944 . SMS bytes freed
backend_req 5696167 21.96 Backend requests made
n_vcl 2 0.00 N vcl total
n_vcl_avail 2 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 709 . N total active bans
n_ban_add 5198 0.02 N new bans added
n_ban_retire 4489 0.02 N old bans deleted
n_ban_obj_test 80926784 312.05 N objects tested
n_ban_re_test 225982892 871.38 N regexps tested against
n_ban_dups 3000 0.01 N duplicate bans removed
hcb_nolock 0 0.00 HCB Lookups without lock
hcb_lock 0 0.00 HCB Lookups with lock
hcb_insert 0 0.00 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 259340 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 3849737 14.84 Gunzip operations
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 267087 1.03 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 27800385 107.20 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 0 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 0 0.00 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 169313 0.65 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 93960449 362.31 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 8668 0.03 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 2580 0.01 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 11156227 43.02 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 2617990 10.09 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 23393 0.09 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 4 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 89338248 344.48 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 3861706 14.89 Created locks
LCK.objhdr.destroy 3338136 12.87 Destroyed locks
LCK.objhdr.locks 108880937 419.84 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 9303270 35.87 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 8143794 31.40 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 31371 0.12 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 90171669 347.70 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 2838069 10.94 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 6589306 25.41 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 19 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 22723286 87.62 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 11969376 46.15 Allocator requests
SMA.s0.c_fail 412116470473 1589097.21 Allocator failures
SMA.s0.c_bytes 277501708539 1070030.49 Bytes allocated
SMA.s0.c_freed 270522525345 1043119.17 Bytes freed
SMA.s0.g_alloc 1096776 . Allocations outstanding
SMA.s0.g_bytes 6979183194 . Bytes outstanding
SMA.s0.g_space 138662 . Bytes available
SMA.Transient.c_req 2763044 10.65 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 179601302623 692532.21 Bytes allocated
SMA.Transient.c_freed 175191674974 675528.94 Bytes freed
SMA.Transient.g_alloc 429519 . Allocations outstanding
SMA.Transient.g_bytes 4409627649 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.fake(127.0.0.1,,80).vcls 2 . VCL references
VBE.fake(127.0.0.1,,80).happy 0 . Happy health probes
VBE.app01(10.140.106.10,,80).vcls 2 . VCL references
VBE.app01(10.140.106.10,,80).happy18446744073709551615 .
Happy health probes
VBE.app02(10.140.106.11,,80).vcls 2 . VCL references
VBE.app02(10.140.106.11,,80).happy18446744073709551615 .
Happy health probes
VBE.app03(10.140.106.12,,80).vcls 2 . VCL references
VBE.app03(10.140.106.12,,80).happy18446744073709551615 .
Happy health probes
VBE.app04(10.140.106.13,,80).vcls 2 . VCL references
VBE.app04(10.140.106.13,,80).happy18446744073709551615 .
Happy health probes
VBE.app05(10.140.106.14,,80).vcls 2 . VCL references
VBE.app05(10.140.106.14,,80).happy18446744073709551615 .
Happy health probes
VBE.app06(10.140.106.15,,80).vcls 2 . VCL references
VBE.app06(10.140.106.15,,80).happy18446744073709551615 .
Happy health probes
VBE.app07(10.140.106.16,,80).vcls 2 . VCL references
VBE.app07(10.140.106.16,,80).happy18446744073709551615 .
Happy health probes
VBE.app08(10.140.106.17,,80).vcls 2 . VCL references
VBE.app08(10.140.106.17,,80).happy18446744073709551615 .
Happy health probes
VBE.app09(10.140.106.39,,80).vcls 2 . VCL references
VBE.app09(10.140.106.39,,80).happy18446744073709551615 .
Happy health probes
VBE.app10(10.140.106.40,,80).vcls 2 . VCL references
VBE.app10(10.140.106.40,,80).happy18446744073709551615 .
Happy health probes
VBE.app11(10.140.106.41,,80).vcls 2 . VCL references
VBE.app11(10.140.106.41,,80).happy18446744073709551615 .
Happy health probes
VBE.app12(10.140.106.42,,80).vcls 2 . VCL references
VBE.app12(10.140.106.42,,80).happy18446744073709551615 .
Happy health probes
VBE.app13(10.140.106.43,,80).vcls 2 . VCL references
VBE.app13(10.140.106.43,,80).happy18446744073709551615 .
Happy health probes
VBE.app14(10.140.106.46,,80).vcls 2 . VCL references
VBE.app14(10.140.106.46,,80).happy18446744073709551615 .
Happy health probes
VBE.static01(10.140.106.18,,8000).vcls 2 . VCL references
VBE.static01(10.140.106.18,,8000).happy18446744073709551615 .
Happy health probes
VBE.static02(10.140.106.19,,8000).vcls 2 . VCL references
VBE.static02(10.140.106.19,,8000).happy18446744073709551615 .
Happy health probes
VBE.static03(10.140.106.20,,8000).vcls 2 . VCL references
VBE.static03(10.140.106.20,,8000).happy18446744073709551615 .
Happy health probes
VBE.static04(10.140.106.21,,8000).vcls 2 . VCL references
VBE.static04(10.140.106.21,,8000).happy18446744073709551615 .
Happy health probes

I'd rather not post the whole VCL - is it possible to assess which
parts would be relevant here?

> My first suspicion is that objects end up in the transient storage.

How would I know?

Thanks
Jan

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Jan 18, 2012, 8:52 AM

Post #9 of 16 (4295 views)
Permalink

Re: Varnish memory usage [In reply to]

On Wed, Jan 18, 2012 at 04:09:09PM +0000, Jan-Aage Frydenb�-Bruvoll wrote:
>
> I'd rather not post the whole VCL - is it possible to assess which
> parts would be relevant here?

Any part that sets ttl.

From man vcl:

shortlived

� Units: s

� Default: 10.0

Objects created with TTL shorter than this are always put in
transient storage.

> > My first suspicion is that objects end up in the transient storage.
>
> How would I know?

From varnishstat:

> SMA.Transient.g_bytes 4409627649 . Bytes outstanding

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

jan at architechs

Jan 22, 2012, 7:32 AM

Post #10 of 16 (4279 views)
Permalink

Re: Varnish memory usage [In reply to]

Hi,

On Wed, Jan 18, 2012 at 16:52, Andreas Plesner Jacobsen <apj [at] mutt> wrote:
> > SMA.Transient.g_bytes 4409627649 . Bytes outstanding

Thanks Andreas, this is incredibly useful information. Now, since last
I posted, the process in question has grown further, to

25653 nobody 20 0 59.7g 21g 81m S 22 22.4 1092:15 varnishd

and varnishstat says

proxy03 jan # varnishstat -1 | grep -i trans
SMA.Transient.c_req 14802020 24.56 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 963605906690 1598878.84 Bytes allocated
SMA.Transient.c_freed 951833551349 1579345.37 Bytes freed
SMA.Transient.g_alloc 1239195 . Allocations outstanding
SMA.Transient.g_bytes 11772355341 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available

When would transient objects actually leave the storage again? Can I
force/trigger/monitor this at all?

The manual says "By default Varnish would use an unlimited malloc
backend for this", which in my case is rather inconvenient since this
varnish process sits in an LXC container, where it apparently sees the
host's memory allocation and not the container's - that in turn leads
to amusing situations where Varnish runs head first into the LXC
resource limitations and gets OOM-killed.

Best regards
Jan

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Jan 22, 2012, 7:44 AM

Post #11 of 16 (4280 views)
Permalink

Re: Varnish memory usage [In reply to]

On Sun, Jan 22, 2012 at 03:32:29PM +0000, Jan-Aage Frydenb�-Bruvoll wrote:
>
> When would transient objects actually leave the storage again? Can I
> force/trigger/monitor this at all?

Same rules apply for transient as for any other malloc store: TTL, LRU or
ban/purge.

> The manual says "By default Varnish would use an unlimited malloc
> backend for this", which in my case is rather inconvenient since this
> varnish process sits in an LXC container, where it apparently sees the
> host's memory allocation and not the container's - that in turn leads
> to amusing situations where Varnish runs head first into the LXC
> resource limitations and gets OOM-killed.

You can just create a storage backend called "Transient" to override the
default. However, I think it's much more appropriate to investigate how they
end up in Transient in the first place.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

jan at architechs

Jan 22, 2012, 7:50 AM

Post #12 of 16 (4278 views)
Permalink

Re: Varnish memory usage [In reply to]

Hi,

On Sun, Jan 22, 2012 at 15:44, Andreas Plesner Jacobsen <apj [at] mutt> wrote:
>
> Same rules apply for transient as for any other malloc store: TTL, LRU or
> ban/purge.

Will Varnish keep multiple generations of objects that expire, and
potentially wait with LRU eviction until it gets closer to the known
sensible maximum memory allocation (the host has 128GB RAM)? We do
have a limited number of (smallish) objects cached for 5 seconds as a
somewhat desperate means of shielding our application servers, but the
amount of pages we cache in this way in no way rhymes with the
enormous amount of data kept by Varnish in transient storage. All
other objects are stored for a substantially longer time (i.e. weeks
and above) and should not come into play here.

Putting things into perspective - the transient storage is currently
almost as big as the full on-disk storage of 6 years of data, and
given that this is a newspaper, 95% of the on-disk data would be
long-tail and hardly ever accessed.

Best regards
Jan

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Jan 22, 2012, 7:54 AM

Post #13 of 16 (4276 views)
Permalink

Re: Varnish memory usage [In reply to]

On Sun, Jan 22, 2012 at 03:50:56PM +0000, Jan-Aage Frydenb�-Bruvoll wrote:
> >
> > Same rules apply for transient as for any other malloc store: TTL, LRU or
> > ban/purge.
>
> Will Varnish keep multiple generations of objects that expire, and
> potentially wait with LRU eviction until it gets closer to the known
> sensible maximum memory allocation (the host has 128GB RAM)? We do
> have a limited number of (smallish) objects cached for 5 seconds as a
> somewhat desperate means of shielding our application servers, but the
> amount of pages we cache in this way in no way rhymes with the
> enormous amount of data kept by Varnish in transient storage. All
> other objects are stored for a substantially longer time (i.e. weeks
> and above) and should not come into play here.

Are you sure that these objects are really hashed correctly, and do not have a
Vary header that will blot the cache? Do you see hits on these objects?

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Jan 22, 2012, 7:58 AM

Post #14 of 16 (4278 views)
Permalink

Re: Varnish memory usage [In reply to]

On Sun, Jan 22, 2012 at 03:50:56PM +0000, Jan-Aage Frydenb�-Bruvoll wrote:
> >
> > Same rules apply for transient as for any other malloc store: TTL, LRU or
> > ban/purge.
>
> Will Varnish keep multiple generations of objects that expire, and
> potentially wait with LRU eviction until it gets closer to the known
> sensible maximum memory allocation (the host has 128GB RAM)?

Sorry, forgot this part: No, Transient is unbounded unless you configure it
otherwise. It should really only be used for short-lived objects.

In your case, I would stick these small objects in a store I controlled the
parameters of (beresp.storage)

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

jan at architechs

Jan 22, 2012, 8:03 AM

Post #15 of 16 (4282 views)
Permalink

Re: Varnish memory usage [In reply to]

Hi,

On Sun, Jan 22, 2012 at 15:54, Andreas Plesner Jacobsen <apj [at] mutt> wrote:
> Are you sure that these objects are really hashed correctly, and do not have a
> Vary header that will blot the cache? Do you see hits on these objects?

I'm never sure of anything, but we've tried to do a good job on
forcing headers down on these objects. However it will certainly not
hurt to go through this again. Also thank you very much for the hint
regarding unbounded transient storage - that's definitely something I
need to look at immediately.

Thanks
Jan

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Jan 22, 2012, 12:33 PM

Post #16 of 16 (4261 views)
Permalink

Re: Varnish memory usage [In reply to]

On Sun, Jan 22, 2012 at 04:58:58PM +0100, Andreas Plesner Jacobsen wrote:
> > >
> > > Same rules apply for transient as for any other malloc store: TTL, LRU or
> > > ban/purge.
> >
> > Will Varnish keep multiple generations of objects that expire, and
> > potentially wait with LRU eviction until it gets closer to the known
> > sensible maximum memory allocation (the host has 128GB RAM)?
>
> Sorry, forgot this part: No, Transient is unbounded unless you configure it
> otherwise. It should really only be used for short-lived objects.

After looking over the code with help from DocWilco, it looks like Transient is
used in three circumstances:

1. Short-lived objects
2. vcl_error
3. When allocations in all other stores fail

It looks like you're hitting 3:
SMA.s0.c_fail 412116470473 1589097.21 Allocator failures

My guess is that you're stressing LRU and are hitting some race conditions
where one request LRUs and makes space for another object, while not actually
making space for itself. I think your only option right now is to be more
aggressive with your TTL and ensure that your main storage has a little space
free.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads