Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Dev

Memory leak in trunk?

 

 

Varnish dev RSS feed   Index | Next | Previous | View Threaded


dmitry.panov at yahoo

Mar 11, 2011, 4:54 AM

Post #1 of 9 (1165 views)
Permalink
Memory leak in trunk?

Hi Geoff, and all,

Well, to me it looks like a memory leak, the virtual memory consumed by
varnishd grew twice as much as the size of the configured storage and
continued to grow until I stopped the test. It doesn't look normal, does
it? Especially considering that there were only a few pages involved.

I tried the same scenario with the latest stable release and it showed a
completely different behavior: Virtual Memory size grew at a much slower
rate and then remained steady even after 300k requests were served.
There is definitely a problem with the current trunk.

Does the output from varnishstat give any clue? What should I post as an
evidence?


On 11/03/2011 07:30, Geoff Simmons wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 3/11/11 2:30 AM, Dmitry Panov wrote:
>> You're right, the patch didn't make any difference. The problem was that
>> I was running an earlier version of the unpatched trunk which contained
>> the ws overflow bug, so the reason memory usage was dropping from time
>> to time was because one of the threads was dying.
>>
>> I've installed the latest trunk and it behaved exactly like the patched
>> version: after about 80k requests the virtual memory size was about 2G
>> (I have 1G storage configured). I ran varnishstat -1 after that (attached).
>>
>> So it looks like there is a memory leak in the current trunk.
> I tried a load test, but couldn't reproduce a memory leak. With the
> latest unpatched trunk using -s malloc,1G, and running httperf and an
> Apache backend all on my machine, I ran a load of about 12,500 reqs/s
> for over a half hour. varnishd's virtual memory size expanded to 911 MB
> after 4 minutes, but no further. The stevedore stats show 415 MB
> outstanding bytes after the run, which is about how much data I have in
> the test site. The stevedore had allocated 7 GB and freed 6.6 GB during
> the course of the run. All of that looks right to me.
>
> Since this is about the trunk itself rather than the patch, and if
> you're sure you can establish that there's a leak, maybe you should file
> a bug report with the evidence.
>
> Thanks again for your help with testing.
>
>
> Best,
> Geoff

Best regards,

--
Dmitry Panov


_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


phk at phk

Mar 11, 2011, 5:03 AM

Post #2 of 9 (1125 views)
Permalink
Re: Memory leak in trunk? [In reply to]

In message <4D7A1B77.1090009 [at] yahoo>, Dmitry Panov writes:

>There is definitely a problem with the current trunk.

is this with or without geoffs ims-patch ?

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


dmitry.panov at yahoo

Mar 11, 2011, 5:04 AM

Post #3 of 9 (1118 views)
Permalink
Re: Memory leak in trunk? [In reply to]

Both. The patch doesn't make any difference.

On 11/03/2011 13:03, Poul-Henning Kamp wrote:
> In message<4D7A1B77.1090009 [at] yahoo>, Dmitry Panov writes:
>
>> There is definitely a problem with the current trunk.
> is this with or without geoffs ims-patch ?
>

Best regards,

--
Dmitry Panov


_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


phk at phk

Mar 11, 2011, 6:06 AM

Post #4 of 9 (1131 views)
Permalink
Re: Memory leak in trunk? [In reply to]

In message <4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:

>Both. The patch doesn't make any difference.

Ok, that makes it my department :-)

Any indication what the leak depends on ?

number of requests ?

size of requests ?

hit rate ?

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


dmitry.panov at yahoo

Mar 11, 2011, 3:45 PM

Post #5 of 9 (1114 views)
Permalink
Re: Memory leak in trunk? [In reply to]

Hi Poul,

I think I found the reason. I tried the same test but with
http_gzip_support=off and it worked fine. tsung does not support gzip so
varnish had to gunzip objects for it and this is where the leak appears
to be. Hope it helps.


On 11/03/2011 14:06, Poul-Henning Kamp wrote:
> In message<4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:
>
>> Both. The patch doesn't make any difference.
> Ok, that makes it my department :-)
>
> Any indication what the leak depends on ?
>
> number of requests ?
>
> size of requests ?
>
> hit rate ?
>


--
Dmitry Panov


_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


perbu at varnish-software

Mar 12, 2011, 12:00 AM

Post #6 of 9 (1108 views)
Permalink
Re: Memory leak in trunk? [In reply to]

Hi.

I've been running trunk on varnish-cache.org since yesterday (14:00) and
there a leak there. Turning off http_gzip_support seems to eliminate the
leak.

Reproducing the leak is quite simple. Just hammer varnish with a client that
doesn't send accept-encoding: gzip.


Per.

On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov <dmitry.panov [at] yahoo>wrote:

> Hi Poul,
>
> I think I found the reason. I tried the same test but with
> http_gzip_support=off and it worked fine. tsung does not support gzip so
> varnish had to gunzip objects for it and this is where the leak appears to
> be. Hope it helps.
>
>
>
> On 11/03/2011 14:06, Poul-Henning Kamp wrote:
>
>> In message<4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:
>>
>> Both. The patch doesn't make any difference.
>>>
>> Ok, that makes it my department :-)
>>
>> Any indication what the leak depends on ?
>>
>> number of requests ?
>>
>> size of requests ?
>>
>> hit rate ?
>>
>>
>
> --
> Dmitry Panov
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev [at] varnish-cache
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>



--
Per Buer, Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Want to learn more about Varnish?
http://www.varnish-software.com/whitepapers


phk at phk

Mar 12, 2011, 6:47 AM

Post #7 of 9 (1140 views)
Permalink
Re: Memory leak in trunk? [In reply to]

In message <AANLkTiknH8QZ7ApZFObmthsQrHp=bLJucKVVi31+b7ea [at] mail>, Per
Buer writes:

I think I've fixed this, a stupid oversight on my part.

Please test again, when possible.

Poul-Henning

>--0015174c3d4addc4d1049e447a1e
>Content-Type: text/plain; charset=ISO-8859-1
>
>Hi.
>
>I've been running trunk on varnish-cache.org since yesterday (14:00) and
>there a leak there. Turning off http_gzip_support seems to eliminate the
>leak.
>
>Reproducing the leak is quite simple. Just hammer varnish with a client that
>doesn't send accept-encoding: gzip.
>
>
>Per.
>
>On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov <dmitry.panov [at] yahoo>wrote:
>
>> Hi Poul,
>>
>> I think I found the reason. I tried the same test but with
>> http_gzip_support=off and it worked fine. tsung does not support gzip so
>> varnish had to gunzip objects for it and this is where the leak appears to
>> be. Hope it helps.
>>
>>
>>
>> On 11/03/2011 14:06, Poul-Henning Kamp wrote:
>>
>>> In message<4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:
>>>
>>> Both. The patch doesn't make any difference.
>>>>
>>> Ok, that makes it my department :-)
>>>
>>> Any indication what the leak depends on ?
>>>
>>> number of requests ?
>>>
>>> size of requests ?
>>>
>>> hit rate ?
>>>
>>>
>>
>> --
>> Dmitry Panov
>>
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev [at] varnish-cache
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
>
>
>--
>Per Buer, Varnish Software
>Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
>Varnish makes websites fly!
>Want to learn more about Varnish?
>http://www.varnish-software.com/whitepapers
>
>--0015174c3d4addc4d1049e447a1e
>Content-Type: text/html; charset=ISO-8859-1
>Content-Transfer-Encoding: quoted-printable
>
>Hi.<div><br></div><div>I&#39;ve been running trunk on <a href=3D"http://var=
>nish-cache.org" target=3D"_blank">varnish-cache.org</a> since yesterday (14=
>:00) and there a leak there. Turning off http_gzip_support seems to elimina=
>te the leak.</div>
>
><div><br></div><div>Reproducing the leak is quite simple. Just hammer varni=
>sh with a client that doesn&#39;t send accept-encoding: gzip.</div><div><br=
>></div><div><br></div><div>Per.<br><br><div class=3D"gmail_quote">On Sat, M=
>ar 12, 2011 at 12:45 AM, Dmitry Panov <span dir=3D"ltr">&lt;<a href=3D"mail=
>to:dmitry.panov [at] yahoo" target=3D"_blank">dmitry.panov [at] yahoo</a>=
>&gt;</span> wrote:<br>
>
>
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
>x #ccc solid;padding-left:1ex">Hi Poul,<br>
><br>
>I think I found the reason. I tried the same test but with http_gzip_suppor=
>t=3Doff and it worked fine. tsung does not support gzip so varnish had to g=
>unzip objects for it and this is where the leak appears to be. Hope it help=
>s.<div>
>
>
><br>
><br>
><br>
>On 11/03/2011 14:06, Poul-Henning Kamp wrote:<br>
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
>x #ccc solid;padding-left:1ex">
>In message&lt;<a href=3D"mailto:4D7A1DC5.7030405 [at] yahoo" target=3D"_bl=
>ank">4D7A1DC5.7030405 [at] yahoo</a>&gt;, Dmitry Panov writes:<br>
><br>
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
>x #ccc solid;padding-left:1ex">
>Both. The patch doesn&#39;t make any difference.<br>
></blockquote>
>Ok, that makes it my department :-)<br>
><br>
>Any indication what the leak depends on ?<br>
><br>
>number of requests ?<br>
><br>
>size of requests ?<br>
><br>
>hit rate ?<br>
><br>
></blockquote>
><br>
><br>
>-- <br></div><font color=3D"#888888">
>Dmitry Panov</font><div><div></div><div><br>
><br>
><br>
>_______________________________________________<br>
>varnish-dev mailing list<br>
><a href=3D"mailto:varnish-dev [at] varnish-cache" target=3D"_blank">varnish-=
>dev [at] varnish-cache</a><br>
><a href=3D"http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev"=
> target=3D"_blank">http://www.varnish-cache.org/lists/mailman/listinfo/varn=
>ish-dev</a><br>
></div></div></blockquote></div><br><br clear=3D"all"><br>-- <br>Per Buer,=
>=A0Varnish Software<br>Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Sk=
>ype: per.buer<br>Varnish makes websites fly!<br>Want to learn more about Va=
>rnish? <a href=3D"http://www.varnish-software.com/whitepapers" target=3D"_b=
>lank">http://www.varnish-software.com/whitepapers</a><br>
>
>
><br>
></div>
>
>--0015174c3d4addc4d1049e447a1e--
>

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


perbu at varnish-software

Mar 12, 2011, 7:08 AM

Post #8 of 9 (1107 views)
Permalink
Re: Memory leak in trunk? [In reply to]

Tested. The leak is gone.

www.varnish-cache.org updated, recompiled and restarted. I've added some ESI
processing to the mix for good measure. :-)

Per.

On Sat, Mar 12, 2011 at 3:47 PM, Poul-Henning Kamp <phk [at] phk>wrote:

> In message <AANLkTiknH8QZ7ApZFObmthsQrHp=bLJucKVVi31+b7ea [at] mail>,
> Per
> Buer writes:
>
> I think I've fixed this, a stupid oversight on my part.
>
> Please test again, when possible.
>
> Poul-Henning
>
> >--0015174c3d4addc4d1049e447a1e
> >Content-Type: text/plain; charset=ISO-8859-1
> >
> >Hi.
> >
> >I've been running trunk on varnish-cache.org since yesterday (14:00) and
> >there a leak there. Turning off http_gzip_support seems to eliminate the
> >leak.
> >
> >Reproducing the leak is quite simple. Just hammer varnish with a client
> that
> >doesn't send accept-encoding: gzip.
> >
> >
> >Per.
> >
> >On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov <dmitry.panov [at] yahoo
> >wrote:
> >
> >> Hi Poul,
> >>
> >> I think I found the reason. I tried the same test but with
> >> http_gzip_support=off and it worked fine. tsung does not support gzip so
> >> varnish had to gunzip objects for it and this is where the leak appears
> to
> >> be. Hope it helps.
> >>
> >>
> >>
> >> On 11/03/2011 14:06, Poul-Henning Kamp wrote:
> >>
> >>> In message<4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:
> >>>
> >>> Both. The patch doesn't make any difference.
> >>>>
> >>> Ok, that makes it my department :-)
> >>>
> >>> Any indication what the leak depends on ?
> >>>
> >>> number of requests ?
> >>>
> >>> size of requests ?
> >>>
> >>> hit rate ?
> >>>
> >>>
> >>
> >> --
> >> Dmitry Panov
> >>
> >>
> >>
> >> _______________________________________________
> >> varnish-dev mailing list
> >> varnish-dev [at] varnish-cache
> >> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
> >>
> >
> >
> >
> >--
> >Per Buer, Varnish Software
> >Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> >Varnish makes websites fly!
> >Want to learn more about Varnish?
> >http://www.varnish-software.com/whitepapers
> >
> >--0015174c3d4addc4d1049e447a1e
> >Content-Type: text/html; charset=ISO-8859-1
> >Content-Transfer-Encoding: quoted-printable
> >
> >Hi.<div><br></div><div>I&#39;ve been running trunk on <a href=3D"
> http://var=
> >nish-cache.org" target=3D"_blank">varnish-cache.org</a> since yesterday
> (14=
> >:00) and there a leak there. Turning off http_gzip_support seems to
> elimina=
> >te the leak.</div>
> >
> ><div><br></div><div>Reproducing the leak is quite simple. Just hammer
> varni=
> >sh with a client that doesn&#39;t send accept-encoding:
> gzip.</div><div><br=
> >></div><div><br></div><div>Per.<br><br><div class=3D"gmail_quote">On Sat,
> M=
> >ar 12, 2011 at 12:45 AM, Dmitry Panov <span dir=3D"ltr">&lt;<a
> href=3D"mail=
> >to:dmitry.panov [at] yahoo" target=3D"_blank">dmitry.panov [at] yahoo
> </a>=
> >&gt;</span> wrote:<br>
> >
> >
> ><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
> .8ex;border-left:1p=
> >x #ccc solid;padding-left:1ex">Hi Poul,<br>
> ><br>
> >I think I found the reason. I tried the same test but with
> http_gzip_suppor=
> >t=3Doff and it worked fine. tsung does not support gzip so varnish had to
> g=
> >unzip objects for it and this is where the leak appears to be. Hope it
> help=
> >s.<div>
> >
> >
> ><br>
> ><br>
> ><br>
> >On 11/03/2011 14:06, Poul-Henning Kamp wrote:<br>
> ><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
> .8ex;border-left:1p=
> >x #ccc solid;padding-left:1ex">
> >In message&lt;<a href=3D"mailto:4D7A1DC5.7030405 [at] yahoo"
> target=3D"_bl=
> >ank">4D7A1DC5.7030405 [at] yahoo</a>&gt;, Dmitry Panov writes:<br>
> ><br>
> ><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
> .8ex;border-left:1p=
> >x #ccc solid;padding-left:1ex">
> >Both. The patch doesn&#39;t make any difference.<br>
> ></blockquote>
> >Ok, that makes it my department :-)<br>
> ><br>
> >Any indication what the leak depends on ?<br>
> ><br>
> >number of requests ?<br>
> ><br>
> >size of requests ?<br>
> ><br>
> >hit rate ?<br>
> ><br>
> ></blockquote>
> ><br>
> ><br>
> >-- <br></div><font color=3D"#888888">
> >Dmitry Panov</font><div><div></div><div><br>
> ><br>
> ><br>
> >_______________________________________________<br>
> >varnish-dev mailing list<br>
> ><a href=3D"mailto:varnish-dev [at] varnish-cache"
> target=3D"_blank">varnish-=
> >dev [at] varnish-cache</a><br>
> ><a href=3D"
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev"=
> > target=3D"_blank">
> http://www.varnish-cache.org/lists/mailman/listinfo/varn=
> >ish-dev</a><br>
> ></div></div></blockquote></div><br><br clear=3D"all"><br>-- <br>Per Buer,=
> >=A0Varnish Software<br>Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 /
> Sk=
> >ype: per.buer<br>Varnish makes websites fly!<br>Want to learn more about
> Va=
> >rnish? <a href=3D"http://www.varnish-software.com/whitepapers"
> target=3D"_b=
> >lank">http://www.varnish-software.com/whitepapers</a><br>
> >
> >
> ><br>
> ></div>
> >
> >--0015174c3d4addc4d1049e447a1e--
> >
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk [at] FreeBSD | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>



--
Per Buer, Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Want to learn more about Varnish?
http://www.varnish-software.com/whitepapers


dmitry.panov at yahoo

Mar 16, 2011, 7:10 AM

Post #9 of 9 (1114 views)
Permalink
Re: Memory leak in trunk? [In reply to]

Hi,

I can also confirm that the leak is gone. Many thanks!

On 12/03/2011 14:47, Poul-Henning Kamp wrote:
> In message<AANLkTiknH8QZ7ApZFObmthsQrHp=bLJucKVVi31+b7ea [at] mail>, Per
> Buer writes:
>
> I think I've fixed this, a stupid oversight on my part.
>
> Please test again, when possible.
>
> Poul-Henning
>
>> --0015174c3d4addc4d1049e447a1e
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> Hi.
>>
>> I've been running trunk on varnish-cache.org since yesterday (14:00) and
>> there a leak there. Turning off http_gzip_support seems to eliminate the
>> leak.
>>
>> Reproducing the leak is quite simple. Just hammer varnish with a client that
>> doesn't send accept-encoding: gzip.
>>
>>
>> Per.
>>
>> On Sat, Mar 12, 2011 at 12:45 AM, Dmitry Panov<dmitry.panov [at] yahoo>wrote:
>>
>>> Hi Poul,
>>>
>>> I think I found the reason. I tried the same test but with
>>> http_gzip_support=off and it worked fine. tsung does not support gzip so
>>> varnish had to gunzip objects for it and this is where the leak appears to
>>> be. Hope it helps.
>>>
>>>
>>>
>>> On 11/03/2011 14:06, Poul-Henning Kamp wrote:
>>>
>>>> In message<4D7A1DC5.7030405 [at] yahoo>, Dmitry Panov writes:
>>>>
>>>> Both. The patch doesn't make any difference.
>>>> Ok, that makes it my department :-)
>>>>
>>>> Any indication what the leak depends on ?
>>>>
>>>> number of requests ?
>>>>
>>>> size of requests ?
>>>>
>>>> hit rate ?
>>>>
>>>>

Best regards,

--
Dmitry Panov


_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Varnish dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.