Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc

varnishd restarting randomly

 

 

Varnish misc RSS feed   Index | Next | Previous | View Threaded


glenn at mp3lyrics

Apr 23, 2012, 2:13 AM

Post #1 of 8 (780 views)
Permalink
varnishd restarting randomly

We've been running varnish on www.mp3lyrics.com for many years,
there has been a problem with varnish restarting at random intervals.
Some times varnish can work for several days before it automatically
restarts (but its rare), and some times it restarts up to 10 times per
day (more "normal").
It does not seem to have anything to do with traffic amount.
This is starting to get annoying and we've reached the point where we
want to either fix the problem with varnish or get another cache
machine.
Last night varnish didn't restart, but it seems it kind of froze. It
didn't fetch data from the webserver, and didn't respond to
http-requsts from the net (this is the first time I've experienced a
varnish freeze like this)
When it froze, all I had to do was to manually restart it, then it
started running "normally" again.

I am unsure whether varnish responds slowly to http-requests from the
net during the automatic restarts or not, but I have come across
situations when varnish is starting to respond slowly, then I restart
it and it starts responding quick again.
This behavior seems totally random, but I have noticed that varnish
can seem slow when very many url.purge commands have been sent to it.
However, the automatic restarting does not seem to have anything to do
with the amount of url.purges sent to varnish (but I can be wrong).

mp3lyrics is a fairly big site, it has several millions (cacheable)
pages, about 200.000 unique users per day, about 300.000 pageviews,
and about 4mill varnish requests per day.
The server is in no way overloaded, apache alone is very capable of
handling the requests, but a cache machine is off course wanted anyway
if it works correctly.
Currently varnish has been given 16GB of ram, I've tried with
different amounts, but the problem seems to be the same.

I need some help to figure out where to start looking for the reason to this.
If someoneĀ could tell me what kind of server type / configuration and
varnish stats / configuration data is needed to start searching for
the reason I'll be happy to provide it.

(or should this request go to the varnish-bugs mailinglist?)

- Glenn-Erik Sandbakken

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


glennerik at glennerik

Apr 23, 2012, 4:05 AM

Post #2 of 8 (751 views)
Permalink
Re: varnishd restarting randomly [In reply to]

Theres nothing about grace in the vcl configuration used, so if grace is
not default we're not using grace.
I have never experienced any problems with other applications/processes
running on www.mp3lyrics.com
If it's due to bad ram then I guess the only solution is switching to a new
server if we wan't to continue using varnish?
Attached is the default.vcl varnish configuration used.

- Glenn-Erik
(ps I changed my varnish mailinglist email address)

2012/4/23 Jonathan Matthews <contact [at] jpluscplusm>:
> Hey Glenn -
>
> I'm not replying on-list as I'm in a rush, not able to give you
> anything definitive, and you sound like you're in a jam,
> operationally.
> Here're my 2 ideas for you from my sysadmin PoV:
>
> 1) Dodgy RAM (or other hardware; but most likely RAM). My #1
> suggestion from what you've described.
>
> 2) Are you using "grace"? It has some undocumented (except on the
> list) flaw where it gobbles more and more and more RAM, eventually
> leading to something going wrong somewhere.
>
> Anyway, HTH.
> Jonathan
>
> On 23 April 2012 10:13, glenn [at] mp3lyrics <glenn [at] mp3lyrics> wrote:
>> We've been running varnish on www.mp3lyrics.com for many years,
>> there has been a problem with varnish restarting at random intervals.
>> Some times varnish can work for several days before it automatically
>> restarts (but its rare), and some times it restarts up to 10 times per
>> day (more "normal").
>> It does not seem to have anything to do with traffic amount.
>> This is starting to get annoying and we've reached the point where we
>> want to either fix the problem with varnish or get another cache
>> machine.
>> Last night varnish didn't restart, but it seems it kind of froze. It
>> didn't fetch data from the webserver, and didn't respond to
>> http-requsts from the net (this is the first time I've experienced a
>> varnish freeze like this)
>> When it froze, all I had to do was to manually restart it, then it
>> started running "normally" again.
>>
>> I am unsure whether varnish responds slowly to http-requests from the
>> net during the automatic restarts or not, but I have come across
>> situations when varnish is starting to respond slowly, then I restart
>> it and it starts responding quick again.
>> This behavior seems totally random, but I have noticed that varnish
>> can seem slow when very many url.purge commands have been sent to it.
>> However, the automatic restarting does not seem to have anything to do
>> with the amount of url.purges sent to varnish (but I can be wrong).
>>
>> mp3lyrics is a fairly big site, it has several millions (cacheable)
>> pages, about 200.000 unique users per day, about 300.000 pageviews,
>> and about 4mill varnish requests per day.
>> The server is in no way overloaded, apache alone is very capable of
>> handling the requests, but a cache machine is off course wanted anyway
>> if it works correctly.
>> Currently varnish has been given 16GB of ram, I've tried with
>> different amounts, but the problem seems to be the same.
>>
>> I need some help to figure out where to start looking for the reason to
this.
>> If someone could tell me what kind of server type / configuration and
>> varnish stats / configuration data is needed to start searching for
>> the reason I'll be happy to provide it.
>>
>> (or should this request go to the varnish-bugs mailinglist?)
>>
>> - Glenn-Erik Sandbakken
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc [at] varnish-cache
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> --
> Jonathan Matthews
> Oxford, London, UK
> http://www.jpluscplusm.com/contact.html
Attachments: default.vcl (6.10 KB)


lasse.karstensen at gmail

Apr 23, 2012, 5:37 AM

Post #3 of 8 (752 views)
Permalink
Re: varnishd restarting randomly [In reply to]

Glenn-Erik Sandbakken:
> Theres nothing about grace in the vcl configuration used, so if grace is
> not default we're not using grace.
> I have never experienced any problems with other applications/processes
> running on www.mp3lyrics.com
> If it's due to bad ram then I guess the only solution is switching to a new
> server if we wan't to continue using varnish?
> Attached is the default.vcl varnish configuration used.

Hi.

What version of Varnish are you running? Are you running distribution
binaries, or something you've compiled yourself?

The management process should log to syslog when the child dies. Can you supply
an example of these lines, please.

Is there any debugging information available when you run "panic.show" in
varnishadm/the management console?

--
Lasse Karstensen
Varnish Software AS


_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


apj at mutt

Apr 23, 2012, 7:03 AM

Post #4 of 8 (752 views)
Permalink
Re: varnishd restarting randomly [In reply to]

On Mon, Apr 23, 2012 at 01:05:49PM +0200, Glenn-Erik Sandbakken wrote:

> > 2) Are you using "grace"? It has some undocumented (except on the
> > list) flaw where it gobbles more and more and more RAM, eventually
> > leading to something going wrong somewhere.

I'd like to hear more about this, since I can't recognize what you're
describing, and if there's a bug, it should be in the tracker.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


glennerik at glennerik

Apr 23, 2012, 8:48 AM

Post #5 of 8 (756 views)
Permalink
Re: varnishd restarting randomly [In reply to]

We're running varnish version 2.0.2 on FreeBSD 7.0
Varnish has been installed with the freebsd ports method, and upgraded once.
It is several years since we upgraded varnish, and it was upgraded in the
effort of fixing this problem, without luck.

There is no panic.show option in the varnishadm console:
*panic.show*
*101 44 *
*Unknown request.*

I will upgrade varnish with the ports method on the server asap, and hope
it remedies the situation.

BTW: The mp3lyrics.com server where varnish is running is a 64Bit XEON with
32GB ram and 8 processors.
The much less powerful server we used prior to the current was a 32Bit XEON
with 8GB ram (using a PAE kernel method to overcome the 4GB address limit)
with 2 processors.
On the previous server we were also running varnish and this was not a
problem !
One of the main reasons for the server switch was to get more ram for
varnish.

Usually varnish operates at very low cpu usage (even though traffic is
high), but some times it uses 100% or more cpu for several minutes.
I'm not sure what varnish is working on during these high cpu usage
periods, but it seems that this is when it sometimes restarts (but not
always, sometimes it reduces the cpu usage without restarting).
In addition, during the high cpu usage periods, sometimes the web response
from varnish on mp3lyrics is slow (seemingly due to varnish working), and
some times it seems quick despite the high cpu usage.
I have no idea for a reason to any of this strange behavior, I used to
think it was normal that varnish periodically used much cpu to create
indexes or whatever, but there's clearly something that is not working
right when it restarts so often.

- Glenn-Erik
http://www.mp3lyrics.com/


2012/4/23 Andreas Plesner Jacobsen <apj [at] mutt>

> On Mon, Apr 23, 2012 at 01:05:49PM +0200, Glenn-Erik Sandbakken wrote:
>
> > > 2) Are you using "grace"? It has some undocumented (except on the
> > > list) flaw where it gobbles more and more and more RAM, eventually
> > > leading to something going wrong somewhere.
>
> I'd like to hear more about this, since I can't recognize what you're
> describing, and if there's a bug, it should be in the tracker.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc [at] varnish-cache
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>


contact at jpluscplusm

Apr 23, 2012, 9:04 AM

Post #6 of 8 (755 views)
Permalink
Re: varnishd restarting randomly [In reply to]

On 23 April 2012 15:03, Andreas Plesner Jacobsen <apj [at] mutt> wrote:
> On Mon, Apr 23, 2012 at 01:05:49PM +0200, Glenn-Erik Sandbakken wrote:
>
>> > 2) Are you using "grace"? It has some undocumented (except on the
>> > list) flaw where it gobbles more and more and more RAM, eventually
>> > leading to something going wrong somewhere.
>
> I'd like to hear more about this, since I can't recognize what you're
> describing, and if there's a bug, it should be in the tracker.

https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-March/021886.html
was the thread I was thinking of.

Jonathan
--
Jonathan Matthews
Oxford, London, UK
http://www.jpluscplusm.com/contact.html

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


apj at mutt

Apr 23, 2012, 9:24 AM

Post #7 of 8 (750 views)
Permalink
Re: varnishd restarting randomly [In reply to]

On Mon, Apr 23, 2012 at 05:04:52PM +0100, Jonathan Matthews wrote:
> >
> >> > 2) Are you using "grace"? It has some undocumented (except on the
> >> > list) flaw where it gobbles more and more and more RAM, eventually
> >> > leading to something going wrong somewhere.
> >
> > I'd like to hear more about this, since I can't recognize what you're
> > describing, and if there's a bug, it should be in the tracker.
>
> https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-March/021886.html
> was the thread I was thinking of.

It's not a flaw as such. It's working as designed: The object is kept in memory
for the duration of grace. Rogier's patch is an optimization where graced
objects can be discarded sooner, if there are alternatives that can cover
delivery of all variants.

I'm not even sure that Per has analyzed the situation correctly. I'm much more
inclined to believe that the reason for the unbounded memory growth is that the
TTL of the objects is less than the shortlived parameter, thus they end up in
the Transient store, which has no limit by default.

So I really believe that mixing grace into this is a wild goose chase.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


glennerik at glennerik

May 9, 2012, 6:36 AM

Post #8 of 8 (693 views)
Permalink
Re: varnishd restarting randomly [In reply to]

I am updating this thread to inform you that after I upgraded to
varnish-3.0.2 revision 55e70a4, about two weeks ago, it's been running
flawless.
Thanks for a great product varnish-guys, keep it up =)

- Glenn-Erik
http://www.mp3lyrics.com/

Varnish misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.