pubcrawler.com at gmail
Oct 4, 2009, 5:57 PM
Post #3 of 3
(306 views)
Permalink
|
|
Re: Open Files in Cherokee - IOCache and Keep Alive
[In reply to]
|
|
Thanks Alvaro, enjoy your much deserved time away; also congratulations.
I am thankful for the information you shared. It confirms what we
are seeing underneath in the OS.
I believe the reverse proxy is fine per se - bug free and not related
to the problems we and some others are having with 503's. Seems ironic
to say that :)
I made some changes to our configuration and am unsure what defaults
for certain values originally should be, but we have Keep Alives now
on as well as IOCACHE enabled and while open descriptors has
increased, it's well within reasonable limits one would expect.
If we make it through Monday uninterrupted by 503's then indeed,
finally, the problem will have been solved and I will provide total
description of the problem and resolution method. At current, we have
touched everything in Cherokee and even some stuff external to
Cherokee, so I am confident of the solution.
I would like to provide the completed debugging, solution and
recommendations to be added into the documentation and/or cookbooks
for future sanity of anyone experiencing the same.
-Paul
On 10/4/09, Alvaro Lopez Ortega <alvaro [at] octality> wrote:
> Hello Paul,
>
> Both I/O-cache and the Keep-alive support are supposed to increase the file
> descriptor usage number. There are good technical reasons for it, actually:
>
> Keep-alive, as its name describes, keeps HTTP connection alive through
> different request sets. In order to do so, the server has to keep a few (or
> a whole bunch, depending on the number of clients) connections opened and
> idle. At that point, two things might happen: the client could send a new
> request using the idle TCP connection, or the server could detect a timeout
> after a few seconds (that time is configurable) and close the connection.
> Cherokee has to keep a file descriptor open for each of those TCP
> connections (client browser -> web server), so if keep-alive is enabled,
> there will be a higher file descriptor usable because of all the idle
> connections; which is the expected behavior.
>
> On the other hand, the I/O-cache has to use a file descriptor to map each
> file into memory. It doesn't read the file in to regular memory, but maps
> it; that's why it needs the file descriptor; I'd have to get into operating
> systems internals in order to explain this properly. But again, I/O-cache
> is expected to use file descriptors as well.
>
> However, as far as we figured, the problem you are suffering must have
> something to do with the reverse proxy. It looks like you hit a file
> descriptor leaking, but that's something we'll have to investigate further.
> (As you probably know, I'm unavailable right now, but I'll try to check it
> out as soon as I'm back..)
>
> By moment, and as a plain work-around, you could add a cron job to
> (gracefully) restart cherokee every few hours. Clients would not notice, and
> the operating system would take care of the file descriptor clean up.
>
> Cheers!
>
> On Mon, Oct 5, 2009 at 3:42 AM, pub crawler <pubcrawler.com [at] gmail>wrote:
>
>> Can anyone explain IOCache and Keep Alive in Cherokee as it relates to
>> open files / file descriptors?
>>
>> We've been looking at output from:
>> lsof -n | grep cherokee | more
>>
>> What we have noticed is that if we enable Keep Alive the output is
>> very long in items. Actually counted over 200 pages of output at one
>> point yesterday.
>>
>> Similarly, IOCache runs the item output count up.
>>
>> Both Keep Alive and IOCache *seem* to have open connections that seem
>> like they should have been terminated long ago. I only say *seem*
>> because this is live on a public site and usage is rather random in
>> such regard.
>>
>> Does "Nonces clean up time" run the task to terminate these
>> abandoned/expired open files? Is there a way to manually or
>> otherwise cleanup all these open files / open descriptors?
>>
>> At current our fix to avoid dreaded 503 errors is to increase ulimit
>> to 90000, turn off IOCache and disable Keep Alive. This is sub-optimal
>> performance and surely wasting resources (power, CPU, etc.).
>>
>> -Paul
>> _______________________________________________
>> Cherokee mailing list
>> Cherokee [at] lists
>> http://lists.octality.com/listinfo/cherokee
>>
>
>
>
> --
> Greetings, alo
> http://www.alobbs.com/
>
_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee
|
