Mailing List Archive: Cherokee: users

Cherokee reaches load of 30

Index | Next | Previous | View Threaded

skinkie at xs4all

Apr 13, 2008, 7:52 AM

Post #1 of 8 (610 views)
Permalink

Cherokee reaches load of 30

Hi,

Today it was the second time Cherokee (0.6.1b1373) had an extremely big
load around 30. Does anyone have an idea how to debug this issue?

Or might know a cause of this failure?

There is currently no logging on the webserver (big traffic eater). I'll
ask if the maintainer can make a graph of the webservers load per 5
minutes or so.

Stefan

_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

adefacc at tin

Apr 14, 2008, 2:21 PM

Post #2 of 8 (578 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

Stefan de Konink wrote:
>
> Hi,
>
> Today it was the second time Cherokee (0.6.1b1373) had an extremely big
> load around 30. Does anyone have an idea how to debug this issue?
>
> Or might know a cause of this failure?

First of all someone should verify what kind of services
are running on that machine.

It could be useful to run a script to poll load level from /proc
or from uptime(1) and when the load goes up it can append to a text file
the list of processes along with the CPU usage,
i.e. (completely untested):

while sleep 5
do
# get load average (only integer part)
LOADAVG=`uptime | cut -d, -f 3 | cut -d: -f 2 | cut -d. -f 2`
# poll load usage

[ $LOADAVG -lt 4 ] && continue

# ok load usage greater than high water mark
# get process usage

# get date and time
date >> file.txt

# get list of open sockets
netstat -a >> file.txt

# get process and CPU usage
top -d 3 -n 1 -b >> file.txt
done

> There is currently no logging on the webserver (big traffic eater). I'll
> ask if the maintainer can make a graph of the webservers load per 5
> minutes or so.

OK, it could also be a DDoS attack, so maybe enabling logging
could clarify what's really happening.

Greetings.

--
Nick Name: A.D.F.
E-Mail: <adefacc () tin ! it>
E-Mail-Format: Plain Text only (please); view using font Courier New
--
_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

skinkie at xs4all

Apr 14, 2008, 2:49 PM

Post #3 of 8 (578 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

On Mon, 14 Apr 2008, A.D.F. wrote:

> First of all someone should verify what kind of services
> are running on that machine.

It is cherokee that has a load of 30.

> It could be useful to run a script to poll load level from /proc
> or from uptime(1) and when the load goes up it can append to a text file
> the list of processes along with the CPU usage,
> i.e. (completely untested):
>
> while sleep 5
> do
> # get load average (only integer part)
> LOADAVG=`uptime | cut -d, -f 3 | cut -d: -f 2 | cut -d. -f 2`
> # poll load usage
>
> [ $LOADAVG -lt 4 ] && continue
>
> # ok load usage greater than high water mark
> # get process usage
>
> # get date and time
> date >> file.txt
>
> # get list of open sockets
> netstat -a >> file.txt
>
> # get process and CPU usage
> top -d 3 -n 1 -b >> file.txt
> done

Currently not an option for me. I also suggested a script in this order.
But they claim that the load from ps aux shows 30 at the cherokee binary.

> > There is currently no logging on the webserver (big traffic eater). I'll
> > ask if the maintainer can make a graph of the webservers load per 5
> > minutes or so.
>
> OK, it could also be a DDoS attack, so maybe enabling logging
> could clarify what's really happening.

A restart of cherokee fixes the problem.

Stefan

_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

adefacc at tin

Apr 14, 2008, 3:27 PM

Post #4 of 8 (578 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

"A.D.F." wrote:
>
> Stefan de Konink wrote:
> >
> > Hi,
> >
> > Today it was the second time Cherokee (0.6.1b1373) had an extremely big
> > load around 30. Does anyone have an idea how to debug this issue?
> >
> > Or might know a cause of this failure?
>
> First of all someone should verify what kind of services
> are running on that machine.
>
> It could be useful to run a script to poll load level from /proc
> or from uptime(1) and when the load goes up it can append to a text file
> the list of processes along with the CPU usage,
> i.e. (completely untested):
>
> while sleep 5
> do
> # get load average (only integer part)
> LOADAVG=`uptime | cut -d, -f 3 | cut -d: -f 2 | cut -d. -f 2`
> # poll load usage

The right instructions are:
LOADAVG=`uptime | cut -d, -f 3 | cut -d: -f 2 | cut -d. -f 1`

--
Nick Name: A.D.F.
E-Mail: <adefacc () tin ! it>
E-Mail-Format: Plain Text only (please); view using font Courier New
--
_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

cespedes at debian

Apr 15, 2008, 3:16 AM

Post #5 of 8 (572 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

On Mon, Apr 14, 2008 at 11:49 PM, Stefan de Konink <skinkie[at]xs4all.nl> wrote:
> It is cherokee that has a load of 30.

Err... what does that mean? There is no such concept (load of a process,
or a group of processes).

Do you mean that you have 30 cherokee threads, each of them running 100%
of the time?

Juan
_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

skinkie at xs4all

Apr 16, 2008, 12:09 PM

Post #6 of 8 (564 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

On Wed, 16 Apr 2008, A.D.F. wrote:

> Sorry for the delay (I have been rather busy in the last 2 days),
> I have a few more questions:
>
> 1) How many times per week does this problem happen ?

From what I saw it takes ~5 days... so it will probably happen again soon.

> 2) Did the previous version of Cherokee trunk (december 2007)
> show the same behaviour ?

Nope, but didn't use max-age then.

> 3) What does strace(1) says (when the load is high) ?
>
> i.e.: strace -ppid [ -ofilename ]
>
> (where pid is the pid of Cherokee process or one of its threads)
>
> also it could be useful to known how many socket connections
> are used by using netstat -a.

I'll ask if it happens again.

> 4) Does the problem go away by using only one thread ?

Not tested.

Stefan

_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

adefacc at tin

Apr 16, 2008, 12:21 PM

Post #7 of 8 (566 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

Stefan de Konink wrote:
>
> On Mon, 14 Apr 2008, A.D.F. wrote:
>
> > First of all someone should verify what kind of services
> > are running on that machine.
>
> It is cherokee that has a load of 30.
>
> > It could be useful to run a script to poll load level from /proc
> > or from uptime(1) and when the load goes up it can append to a text file
> > the list of processes along with the CPU usage,
> > i.e. (completely untested):
> >
> > while sleep 5
> > do
> > # get load average (only integer part)
> > LOADAVG=`uptime | cut -d, -f 3 | cut -d: -f 2 | cut -d. -f 2`
> > # poll load usage
> >
> > [ $LOADAVG -lt 4 ] && continue
> >
> > # ok load usage greater than high water mark
> > # get process usage
> >
> > # get date and time
> > date >> file.txt
> >
> > # get list of open sockets
> > netstat -a >> file.txt
> >
> > # get process and CPU usage
> > top -d 3 -n 1 -b >> file.txt
> > done
>
> Currently not an option for me. I also suggested a script in this order.
> But they claim that the load from ps aux shows 30 at the cherokee binary.
> .
> .
> .
> A restart of cherokee fixes the problem.

Sorry for the delay (I have been rather busy in the last 2 days),
I have a few more questions:

1) How many times per week does this problem happen ?

2) Did the previous version of Cherokee trunk (december 2007)
show the same behaviour ?

3) What does strace(1) says (when the load is high) ?

i.e.: strace -ppid [ -ofilename ]

(where pid is the pid of Cherokee process or one of its threads)

also it could be useful to known how many socket connections
are used by using netstat -a.

4) Does the problem go away by using only one thread ?

Meanwhile within 2-3 days I'll review source code.

--
Nick Name: A.D.F.
E-Mail: <adefacc () tin ! it>
E-Mail-Format: Plain Text only (please); view using font Courier New
--
_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

alvaro at sun

Apr 16, 2008, 10:33 PM

Post #8 of 8 (564 views)
Permalink

Re: Cherokee reaches load of 3030 [In reply to]

On 16 Apr 2008, at 21:09, Stefan de Konink wrote:
> On Wed, 16 Apr 2008, A.D.F. wrote:
>
>> Sorry for the delay (I have been rather busy in the last 2 days),
>> I have a few more questions:
>>
>> 1) How many times per week does this problem happen ?
>
> From what I saw it takes ~5 days... so it will probably happen again
> soon.

I will to reproduce it by making a few million request to a server..
hopefully we won't need to wait for 5 days to reproduce it.

>> 2) Did the previous version of Cherokee trunk (december 2007)
>> show the same behaviour ?
>
> Nope, but didn't use max-age then.

I would bet max-age has nothing to do. That was an almost trivial
change where most of the changes were only for the configuration
methods.

>> 3) What does strace(1) says (when the load is high) ?
>>
>> i.e.: strace -ppid [ -ofilename ]
>>
>> (where pid is the pid of Cherokee process or one of its threads)
>>
>> also it could be useful to known how many socket connections
>> are used by using netstat -a.
>
> I'll ask if it happens again.
>
>> 4) Does the problem go away by using only one thread ?
>
> Not tested.

Yep, I do not think there are many people running single-thread
cherokees in non-embedded devices. :-)

--
Greetings, alo.

_______________________________________________
Cherokee mailing list
Cherokee[at]cherokee-project.com
http://cherokee-project.com/cgi-bin/mailman/listinfo/cherokee

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com