Mailing List Archive: Cherokee: users

Question about open sockets

Index | Next | Previous | View Threaded

reidrac at usebox

Jan 18, 2010, 4:47 AM

Post #1 of 12 (688 views)
Permalink

Question about open sockets

Hello,

After some time running (with low load), when I check the files open by
the cherokee user (www) with fstat (this is a BSD system, it's like
lsof), I get:

http://pastebin.ubuntu.com/358471/

According to man page: "A `*' is used to indicate an INADDR_ANY
binding.".

When I check netstat, I can't see open connections, so I'm wondering why
is Cherokee keeping open so many files, and if it's normal or may be
it's a bug.

Sometimes the number of opened socked decreases a bit, but usually it
grows.

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

alvaro at octality

Jan 18, 2010, 5:01 AM

Post #2 of 12 (665 views)
Permalink

Re: Question about open sockets [In reply to]

On 18/01/2010, at 13:47, Juan J. Martínez wrote:

> After some time running (with low load), when I check the files open by
> the cherokee user (www) with fstat (this is a BSD system, it's like
> lsof), I get:
>
> http://pastebin.ubuntu.com/358471/
>
> According to man page: "A `*' is used to indicate an INADDR_ANY
> binding.".
>
> When I check netstat, I can't see open connections, so I'm wondering why
> is Cherokee keeping open so many files, and if it's normal or may be
> it's a bug.
>
> Sometimes the number of opened socked decreases a bit, but usually it
> grows.

Those open sockets are most likely keep-alive connections that the client browser keeps open for a while.

You could try to turn off keep-alive support in the server, just like a test to check whether those open sockets disappear.

--
Octality
http://www.octality.com/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 18, 2010, 6:16 AM

Post #3 of 12 (666 views)
Permalink

Re: Question about open sockets [In reply to]

El lun, 18-01-2010 a las 14:01 +0100, Alvaro Lopez Ortega escribiÃ³:
> On 18/01/2010, at 13:47, Juan J. MartÃnez wrote:
>
> > After some time running (with low load), when I check the files open by
> > the cherokee user (www) with fstat (this is a BSD system, it's like
> > lsof), I get:
> >
> > http://pastebin.ubuntu.com/358471/
> >
> > According to man page: "A `*' is used to indicate an INADDR_ANY
> > binding.".
> >
> > When I check netstat, I can't see open connections, so I'm wondering why
> > is Cherokee keeping open so many files, and if it's normal or may be
> > it's a bug.
> >
> > Sometimes the number of opened socked decreases a bit, but usually it
> > grows.
>
> Those open sockets are most likely keep-alive connections that the client browser keeps open for a while.
>
> You could try to turn off keep-alive support in the server, just like a test to check whether those open sockets disappear.

But those keep-alive connections, will them appear in the netstat
output, won't them?

Right now cherokee has 137 open files, and 123 are '*'. I don't believe
I have 123 keep-alive connections.

I'm going to switch the feature off and see what happens.

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

stefan at konink

Jan 18, 2010, 6:19 AM

Post #4 of 12 (661 views)
Permalink

Re: Question about open sockets [In reply to]

On Mon, 18 Jan 2010, Juan J. Martínez wrote:

> But those keep-alive connections, will them appear in the netstat
> output, won't them?
>
> Right now cherokee has 137 open files, and 123 are '*'. I don't believe
> I have 123 keep-alive connections.
>
> I'm going to switch the feature off and see what happens.

I have had this too with the development of handler_tile, it seems that
connections closed by cherokee, are infact still open on the other side.
That basically created all kind of fds starvation.

Stefan

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 18, 2010, 6:30 AM

Post #5 of 12 (667 views)
Permalink

Re: Question about open sockets [In reply to]

El lun, 18-01-2010 a las 15:19 +0100, Stefan de Konink escribiÃ³:
> On Mon, 18 Jan 2010, Juan J. MartÃnez wrote:
>
> > But those keep-alive connections, will them appear in the netstat
> > output, won't them?
> >
> > Right now cherokee has 137 open files, and 123 are '*'. I don't believe
> > I have 123 keep-alive connections.
> >
> > I'm going to switch the feature off and see what happens.
>
> I have had this too with the development of handler_tile, it seems that
> connections closed by cherokee, are infact still open on the other side.
> That basically created all kind of fds starvation.

But I insist: if there's an open connection, it must appear in netstat,
and it's not the case here.

I've disabled the keep alive for two vhost with some traffic. One serves
a PHP site and the other one is a HTTP proxy.

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 18, 2010, 6:51 AM

Post #6 of 12 (666 views)
Permalink

Re: Question about open sockets [In reply to]

El lun, 18-01-2010 a las 15:30 +0100, Juan J. MartÃnez escribiÃ³:
> [...]
> But I insist: if there's an open connection, it must appear in netstat,
> and it's not the case here.
>
> I've disabled the keep alive for two vhost with some traffic. One serves
> a PHP site and the other one is a HTTP proxy.

OK, it looks light it might be related to the http proxy and the
keep-alive between the http proxy requests and the proxyfied
application.

And the weird thing is that it looks like a fd leak, because there isn't
a open connection associated to that fd (according to netstat).

I'll try to peep into handler_proxy.c tonight.

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

alvaro at octality

Jan 18, 2010, 7:08 AM

Post #7 of 12 (670 views)
Permalink

Re: Question about open sockets [In reply to]

On 18/01/2010, at 15:51, Juan J. Martínez wrote:
> El lun, 18-01-2010 a las 15:30 +0100, Juan J. Martínez escribió:
>> [...]
>> But I insist: if there's an open connection, it must appear in netstat,
>> and it's not the case here.
>>
>> I've disabled the keep alive for two vhost with some traffic. One serves
>> a PHP site and the other one is a HTTP proxy.
>
> OK, it looks light it might be related to the http proxy and the
> keep-alive between the http proxy requests and the proxyfied
> application.
>
> And the weird thing is that it looks like a fd leak, because there isn't
> a open connection associated to that fd (according to netstat).

The proxy maintains a poll of open connections. If the number of descriptors does not grow I'd assume that's the reason.

> I'll try to peep into handler_proxy.c tonight.

Grand! Good luck.. :-)

--
Octality
http://www.octality.com/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 18, 2010, 7:43 AM

Post #8 of 12 (662 views)
Permalink

Re: Question about open sockets [In reply to]

El lun, 18-01-2010 a las 16:08 +0100, Alvaro Lopez Ortega escribiÃ³:
> On 18/01/2010, at 15:51, Juan J. MartÃnez wrote:
> > El lun, 18-01-2010 a las 15:30 +0100, Juan J. MartÃnez escribiÃ³:
> >> [...]
> >> But I insist: if there's an open connection, it must appear in netstat,
> >> and it's not the case here.
> >>
> >> I've disabled the keep alive for two vhost with some traffic. One serves
> >> a PHP site and the other one is a HTTP proxy.
> >
> > OK, it looks light it might be related to the http proxy and the
> > keep-alive between the http proxy requests and the proxyfied
> > application.
> >
> > And the weird thing is that it looks like a fd leak, because there isn't
> > a open connection associated to that fd (according to netstat).
>
> The proxy maintains a poll of open connections. If the number of descriptors does not grow I'd assume that's the reason.
>
> > I'll try to peep into handler_proxy.c tonight.
>
> Grand! Good luck.. :-)

Hello,

I did a quick test:

1. Ask for a proxyfied page
2. Wait ~15 secs (it's the keep-alive timeout for the proxyfied app)
3. Check netstat: shows CLOSE_WAIT/FIN_WAIT_2 state for the connections
(one for cherockee and the other for the proxyfied app, respectively)
4. fstat shows the socket is open (we know it's not true, according to
netstat!)
5. Ask for another proxyfied page
6. Here you are! A new wasted socket! (I guess it's the previous one)

A can reproduce it every time.

The question is: is this the correct behavior?

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

alvaro at octality

Jan 20, 2010, 2:07 AM

Post #9 of 12 (660 views)
Permalink

Re: Question about open sockets [In reply to]

On 18/01/2010, at 16:43, Juan J. Martínez wrote:
> El lun, 18-01-2010 a las 16:08 +0100, Alvaro Lopez Ortega escribió:
>> On 18/01/2010, at 15:51, Juan J. Martínez wrote:
>>> El lun, 18-01-2010 a las 15:30 +0100, Juan J. Martínez escribió:
>>>> [...]
>>>> But I insist: if there's an open connection, it must appear in netstat,
>>>> and it's not the case here.
>>>>
>>>> I've disabled the keep alive for two vhost with some traffic. One serves
>>>> a PHP site and the other one is a HTTP proxy.
>>>
>>> OK, it looks light it might be related to the http proxy and the
>>> keep-alive between the http proxy requests and the proxyfied
>>> application.
>>>
>>> And the weird thing is that it looks like a fd leak, because there isn't
>>> a open connection associated to that fd (according to netstat).
>>
>> The proxy maintains a poll of open connections. If the number of descriptors does not grow I'd assume that's the reason.
>>
>>> I'll try to peep into handler_proxy.c tonight.
>
> I did a quick test:
>
> 1. Ask for a proxyfied page
> 2. Wait ~15 secs (it's the keep-alive timeout for the proxyfied app)
> 3. Check netstat: shows CLOSE_WAIT/FIN_WAIT_2 state for the connections
> (one for cherockee and the other for the proxyfied app, respectively)
> 4. fstat shows the socket is open (we know it's not true, according to
> netstat!)

Let's say it's being closed. It's between both states.

> 5. Ask for another proxyfied page
> 6. Here you are! A new wasted socket! (I guess it's the previous one)

Does it turn into a file descriptor leak if you request a few hundred pages?

> A can reproduce it every time.
> The question is: is this the correct behavior?

Cherokee's proxy keeps a connection poll. The connections in that poll are supposed to be open, so the server can reuse then and save a three-way TCP handshake. However, it has no way to know whether those connections are actually open or if the remote peer has performed a socket shutdown (that's where the FIN_WAIT_2 state comes from).

Due to this situation, Cherokee supposes the connections to be open. In case a connection were closed, it'd simply close it and open a new one. If there were a file descriptor leak, this would be most likely the buggy piece of code.

But before I continue speculating, it'd be important to know whether the leak actually exist. Does making a hundred thousand requests to the proxy make any difference? Are more descriptors being left opened?

--
Octality
http://www.octality.com/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 20, 2010, 3:09 AM

Post #10 of 12 (650 views)
Permalink

Re: Question about open sockets [In reply to]

El miÃ©, 20-01-2010 a las 11:07 +0100, Alvaro Lopez Ortega escribiÃ³:
> [...]
> But before I continue speculating, it'd be important to know whether the leak actually exist. Does making a hundred thousand requests to the proxy make any difference? Are more descriptors being left opened?

I'd have liked to peep in the proxy code, but I haven't had the time
yet :(

Anyway, I did a test following the steps to reproduce the problem:

1. Get a URL through the proxy
2. Wait for timeout
3. Go to 1

The script for the test is here: http://gist.github.com/281767

At the end it shows the difference in open fds according to fstat
(lsof), and my results for 100 loops are:

URL: ***, Loops: 100 -- FD Before: 79, After: 177, Diff: 98

It can be easily ported to Linux using lsof instead of fstat.

Please notice that the server it's in production, so the results may
have some noise for real requests (but not very much, because it's a
very low traffic site).

I don't know if that's OK, or it's related to some dark behavior in
OpenBSD, or if I'm totally wrong :D

Regards,

Juanjo

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

reidrac at usebox

Jan 23, 2010, 12:51 PM

Post #11 of 12 (651 views)
Permalink

Re: Question about open sockets [In reply to]

Hello,

I can confirm there's a fd leak in some conditions in the proxy handler.

When the keep-alive time arrives in the proxified application, the
socket it's closed in the proxified application. Cherokee keeps this
connection cached to reuse it in next calls (as expected).

A new request arrives, and then cherokee_socket_is_connected gets a
ret_deny error, sets respined to true and tries to reconnect.

The problem is the socket is cleaned *before* closing, thus the close
call does nothing and we have a fd leak :(

I've repeated the fdtest [1] with the attached patch applied (it's just
swapping the clean and close calls in the reconnection with respin
enabled :D).

Before the patch:

URL: ***, Loops: 100 -- FD Before: 79, After: 177, Diff: 98

After the patch:

URL: ***, Loops: 100 -- FD Before: 15, After: 14, Diff: -1

Keep in mind that the site it's in production, so the test results can
have some noise.

Although the change seems pretty obvious, I'm newbie with the code. The
test confirms the patch, though.

Regards,

Juanjo

[1]: http://gist.github.com/281767

El miÃ©, 20-01-2010 a las 12:09 +0100, Juan J. MartÃnez escribiÃ³:
> El miÃ©, 20-01-2010 a las 11:07 +0100, Alvaro Lopez Ortega escribiÃ³:
> > [...]
> > But before I continue speculating, it'd be important to know whether the leak actually exist. Does making a hundred thousand requests to the proxy make any difference? Are more descriptors being left opened?
>
> I'd have liked to peep in the proxy code, but I haven't had the time
> yet :(
>
> Anyway, I did a test following the steps to reproduce the problem:
>
> 1. Get a URL through the proxy
> 2. Wait for timeout
> 3. Go to 1
>
> The script for the test is here: http://gist.github.com/281767
>
> At the end it shows the difference in open fds according to fstat
> (lsof), and my results for 100 loops are:
>
> URL: ***, Loops: 100 -- FD Before: 79, After: 177, Diff: 98
>
> It can be easily ported to Linux using lsof instead of fstat.
>
> Please notice that the server it's in production, so the results may
> have some noise for real requests (but not very much, because it's a
> very low traffic site).
>
> I don't know if that's OK, or it's related to some dark behavior in
> OpenBSD, or if I'm totally wrong :D
>
> Regards,
>
> Juanjo
>

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/
ramble on: http://rambleon.usebox.net/

Attachments:

handler_proxy_leak.patch (0.49 KB)

alvaro at octality

Jan 23, 2010, 2:15 PM

Post #12 of 12 (643 views)
Permalink

Re: Question about open sockets [In reply to]

Hello there,

You've certainly found a nasty bug. Great catch!!

I'm applying your patch right away.
Good stuff! :-)

On 23/01/2010, at 21:51, Juan J. Martínez wrote:

> I can confirm there's a fd leak in some conditions in the proxy handler.
>
> When the keep-alive time arrives in the proxified application, the
> socket it's closed in the proxified application. Cherokee keeps this
> connection cached to reuse it in next calls (as expected).
>
> A new request arrives, and then cherokee_socket_is_connected gets a
> ret_deny error, sets respined to true and tries to reconnect.
>
> The problem is the socket is cleaned *before* closing, thus the close
> call does nothing and we have a fd leak :(
>
> I've repeated the fdtest [1] with the attached patch applied (it's just
> swapping the clean and close calls in the reconnection with respin
> enabled :D).
>
> Before the patch:
>
> URL: ***, Loops: 100 -- FD Before: 79, After: 177, Diff: 98
>
> After the patch:
>
> URL: ***, Loops: 100 -- FD Before: 15, After: 14, Diff: -1
>
> Keep in mind that the site it's in production, so the test results can
> have some noise.
>
> Although the change seems pretty obvious, I'm newbie with the code. The
> test confirms the patch, though.
>
> Regards,
>
> Juanjo
>
> [1]: http://gist.github.com/281767
>
> El mié, 20-01-2010 a las 12:09 +0100, Juan J. Martínez escribió:
>> El mié, 20-01-2010 a las 11:07 +0100, Alvaro Lopez Ortega escribió:
>>> [...]
>>> But before I continue speculating, it'd be important to know whether the leak actually exist. Does making a hundred thousand requests to the proxy make any difference? Are more descriptors being left opened?
>>
>> I'd have liked to peep in the proxy code, but I haven't had the time
>> yet :(
>>
>> Anyway, I did a test following the steps to reproduce the problem:
>>
>> 1. Get a URL through the proxy
>> 2. Wait for timeout
>> 3. Go to 1
>>
>> The script for the test is here: http://gist.github.com/281767
>>
>> At the end it shows the difference in open fds according to fstat
>> (lsof), and my results for 100 loops are:
>>
>> URL: ***, Loops: 100 -- FD Before: 79, After: 177, Diff: 98
>>
>> It can be easily ported to Linux using lsof instead of fstat.
>>
>> Please notice that the server it's in production, so the results may
>> have some noise for real requests (but not very much, because it's a
>> very low traffic site).
>>
>> I don't know if that's OK, or it's related to some dark behavior in
>> OpenBSD, or if I'm totally wrong :D
>>
>> Regards,
>>
>> Juanjo
>>
>
> --
> jjm's home: http://www.usebox.net/jjm/
> blackshell: http://blackshell.usebox.net/
> ramble on: http://rambleon.usebox.net/
>
> <handler_proxy_leak.patch>

--
Octality
http://www.octality.com/

_______________________________________________
Cherokee mailing list
Cherokee [at] lists
http://lists.octality.com/listinfo/cherokee

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads