Mailing List Archive: Apache: Dev

mod_proxy, oooled backend connections and the keep-alive race condition

Index | Next | Previous | View Threaded

thomas.r.w.eckert at gmail

Aug 2, 2013, 2:28 AM

Post #1 of 15 (76 views)
Permalink

mod_proxy, oooled backend connections and the keep-alive race condition

So I've been seeing lots of "proxy: error reading status line from remote
server" by mod_proxy lately. Usually this is caused by the race condition
between checking the connection state and the backend closing the
connection due to the keep-alive timeout. As Covener pointed out to me in
IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does
offer a solution to the problem albeit at the cost of performance.

The call to ap_proxy_http_process_response() in mod_proxy_http.c eventually
boils down to ap_rgetline_core() which calls ap_get_brigade() on
r->input_filters. This looks to me like a simple input filter might do the
trick if it only checked for a possibility to read on the socket and
reopens the connection upon failure type "reset by peer". I took a short
look at core_create_proxy_req() in server/core.c to see how connections are
set up and I wonder if it's possible to recreate/reuse that logic in an
input filter. If so, this input filter would offer a nice alternative if
hard coding this behavior into mod_proxy/core is frowned upon. Simply make
the filter dependant on an env variable, just like proxy-initial-not-pooled.

jim at jaguNET

Aug 2, 2013, 5:28 AM

Post #2 of 15 (74 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

+1 for the theory, but I'm not sure if it's feasible or not.

On Aug 2, 2013, at 5:28 AM, Thomas Eckert <thomas.r.w.eckert [at] gmail> wrote:

> So I've been seeing lots of "proxy: error reading status line from remote server" by mod_proxy lately. Usually this is caused by the race condition between checking the connection state and the backend closing the connection due to the keep-alive timeout. As Covener pointed out to me in IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does offer a solution to the problem albeit at the cost of performance.
>
> The call to ap_proxy_http_process_response() in mod_proxy_http.c eventually boils down to ap_rgetline_core() which calls ap_get_brigade() on r->input_filters. This looks to me like a simple input filter might do the trick if it only checked for a possibility to read on the socket and reopens the connection upon failure type "reset by peer". I took a short look at core_create_proxy_req() in server/core.c to see how connections are set up and I wonder if it's possible to recreate/reuse that logic in an input filter. If so, this input filter would offer a nice alternative if hard coding this behavior into mod_proxy/core is frowned upon. Simply make the filter dependant on an env variable, just like proxy-initial-not-pooled.
>

ruediger.pluem at vodafone

Aug 2, 2013, 5:33 AM

Post #3 of 15 (73 views)
Permalink

AW: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below.

Regards

Rüdiger

> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Freitag, 2. August 2013 14:29
> An: dev [at] httpd
> Betreff: Re: mod_proxy, oooled backend connections and the keep-alive
> race condition
>
> +1 for the theory, but I'm not sure if it's feasible or not.
>
> On Aug 2, 2013, at 5:28 AM, Thomas Eckert <thomas.r.w.eckert [at] gmail>
> wrote:
>
> > So I've been seeing lots of "proxy: error reading status line from
> remote server" by mod_proxy lately. Usually this is caused by the race
> condition between checking the connection state and the backend closing
> the connection due to the keep-alive timeout. As Covener pointed out to
> me in IRC, using mod_proxy_http's env variable "proxy-initial-not-
> pooled" does offer a solution to the problem albeit at the cost of
> performance.
> >
> > The call to ap_proxy_http_process_response() in mod_proxy_http.c
> eventually boils down to ap_rgetline_core() which calls ap_get_brigade()
> on r->input_filters. This looks to me like a simple input filter might
> do the trick if it only checked for a possibility to read on the socket
> and reopens the connection upon failure type "reset by peer". I took a
> short look at core_create_proxy_req() in server/core.c to see how
> connections are set up and I wonder if it's possible to recreate/reuse
> that logic in an input filter. If so, this input filter would offer a
> nice alternative if hard coding this behavior into mod_proxy/core is
> frowned upon. Simply make the filter dependant on an env variable, just
> like proxy-initial-not-pooled.
> >

thomas.r.w.eckert at gmail

Aug 2, 2013, 5:35 AM

Post #4 of 15 (73 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

Yes, the theory thing ... I wish I could have added an experimental patch
for such an input filter but I'm afraid that might take a long time for me
to finish. I'll try though I hope someone more knowledgeable will pick this
up.

On Fri, Aug 2, 2013 at 2:28 PM, Jim Jagielski <jim [at] jagunet> wrote:

> +1 for the theory, but I'm not sure if it's feasible or not.
>
> On Aug 2, 2013, at 5:28 AM, Thomas Eckert <thomas.r.w.eckert [at] gmail>
> wrote:
>
> > So I've been seeing lots of "proxy: error reading status line from
> remote server" by mod_proxy lately. Usually this is caused by the race
> condition between checking the connection state and the backend closing the
> connection due to the keep-alive timeout. As Covener pointed out to me in
> IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does
> offer a solution to the problem albeit at the cost of performance.
> >
> > The call to ap_proxy_http_process_response() in mod_proxy_http.c
> eventually boils down to ap_rgetline_core() which calls ap_get_brigade() on
> r->input_filters. This looks to me like a simple input filter might do the
> trick if it only checked for a possibility to read on the socket and
> reopens the connection upon failure type "reset by peer". I took a short
> look at core_create_proxy_req() in server/core.c to see how connections are
> set up and I wonder if it's possible to recreate/reuse that logic in an
> input filter. If so, this input filter would offer a nice alternative if
> hard coding this behavior into mod_proxy/core is frowned upon. Simply make
> the filter dependant on an env variable, just like proxy-initial-not-pooled.
> >
>
>

jim at jaguNET

Aug 2, 2013, 6:20 AM

Post #5 of 15 (73 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

True enough, but that's inelegant ;)

On Aug 2, 2013, at 8:33 AM, Plüm, Rüdiger, Vodafone Group <ruediger.pluem [at] vodafone> wrote:

> The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below.
>
> Regards
>
> Rüdiger
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jim Jagielski
>> Gesendet: Freitag, 2. August 2013 14:29
>> An: dev [at] httpd
>> Betreff: Re: mod_proxy, oooled backend connections and the keep-alive
>> race condition
>>
>> +1 for the theory, but I'm not sure if it's feasible or not.
>>
>> On Aug 2, 2013, at 5:28 AM, Thomas Eckert <thomas.r.w.eckert [at] gmail>
>> wrote:
>>
>>> So I've been seeing lots of "proxy: error reading status line from
>> remote server" by mod_proxy lately. Usually this is caused by the race
>> condition between checking the connection state and the backend closing
>> the connection due to the keep-alive timeout. As Covener pointed out to
>> me in IRC, using mod_proxy_http's env variable "proxy-initial-not-
>> pooled" does offer a solution to the problem albeit at the cost of
>> performance.
>>>
>>> The call to ap_proxy_http_process_response() in mod_proxy_http.c
>> eventually boils down to ap_rgetline_core() which calls ap_get_brigade()
>> on r->input_filters. This looks to me like a simple input filter might
>> do the trick if it only checked for a possibility to read on the socket
>> and reopens the connection upon failure type "reset by peer". I took a
>> short look at core_create_proxy_req() in server/core.c to see how
>> connections are set up and I wonder if it's possible to recreate/reuse
>> that logic in an input filter. If so, this input filter would offer a
>> nice alternative if hard coding this behavior into mod_proxy/core is
>> frowned upon. Simply make the filter dependant on an env variable, just
>> like proxy-initial-not-pooled.
>>>
>

ruediger.pluem at vodafone

Aug 2, 2013, 6:23 AM

Post #6 of 15 (73 views)
Permalink

AW: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

But effective and simple :-)

> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Freitag, 2. August 2013 15:20
> An: dev [at] httpd
> Betreff: Re: mod_proxy, oooled backend connections and the keep-alive
> race condition
>
> True enough, but that's inelegant ;)
>
> On Aug 2, 2013, at 8:33 AM, Plüm, Rüdiger, Vodafone Group
> <ruediger.pluem [at] vodafone> wrote:
>
> > The typical way to solve this today is to know the keepalive timeout
> of the backend and set ttl for this worker to a value a few seconds
> below.
> >
> > Regards
> >
> > Rüdiger
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Jim Jagielski
> >> Gesendet: Freitag, 2. August 2013 14:29
> >> An: dev [at] httpd
> >> Betreff: Re: mod_proxy, oooled backend connections and the keep-alive
> >> race condition
> >>
> >> +1 for the theory, but I'm not sure if it's feasible or not.
> >>
> >> On Aug 2, 2013, at 5:28 AM, Thomas Eckert
> <thomas.r.w.eckert [at] gmail>
> >> wrote:
> >>
> >>> So I've been seeing lots of "proxy: error reading status line from
> >> remote server" by mod_proxy lately. Usually this is caused by the
> race
> >> condition between checking the connection state and the backend
> closing
> >> the connection due to the keep-alive timeout. As Covener pointed out
> to
> >> me in IRC, using mod_proxy_http's env variable "proxy-initial-not-
> >> pooled" does offer a solution to the problem albeit at the cost of
> >> performance.
> >>>
> >>> The call to ap_proxy_http_process_response() in mod_proxy_http.c
> >> eventually boils down to ap_rgetline_core() which calls
> ap_get_brigade()
> >> on r->input_filters. This looks to me like a simple input filter
> might
> >> do the trick if it only checked for a possibility to read on the
> socket
> >> and reopens the connection upon failure type "reset by peer". I took
> a
> >> short look at core_create_proxy_req() in server/core.c to see how
> >> connections are set up and I wonder if it's possible to
> recreate/reuse
> >> that logic in an input filter. If so, this input filter would offer a
> >> nice alternative if hard coding this behavior into mod_proxy/core is
> >> frowned upon. Simply make the filter dependant on an env variable,
> just
> >> like proxy-initial-not-pooled.
> >>>
> >

covener at gmail

Aug 2, 2013, 6:47 AM

Post #7 of 15 (73 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

>> >> +1 for the theory, but I'm not sure if it's feasible or not.

Some other ideas:

* Does something like a select/poll/getpeername() ever tell us of an
error when a write() doesn't?
* Could we give people an Expect: 100-continue option on the initial
client request?

ruediger.pluem at vodafone

Aug 2, 2013, 7:16 AM

Post #8 of 15 (73 views)
Permalink

AW: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

> -----Ursprüngliche Nachricht-----
> Von: Eric Covener [mailto:covener [at] gmail]
> Gesendet: Freitag, 2. August 2013 15:47
> An: dev [at] httpd
> Betreff: Re: mod_proxy, oooled backend connections and the keep-alive
> race condition
>
> >> >> +1 for the theory, but I'm not sure if it's feasible or not.
>
> Some other ideas:
>
> * Does something like a select/poll/getpeername() ever tell us of an
> error when a write() doesn't?

I don't think so. See the code where we detect if the connection is still there (is_connected ???)

> * Could we give people an Expect: 100-continue option on the initial
> client request?

I think this does not work for GET requests or request without a request body.

Regards

Rüdiger

covener at gmail

Aug 2, 2013, 8:21 AM

Post #9 of 15 (73 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

> I think this does not work for GET requests or request without a request body.

Just re-read spec, you are right -- we are abusing this in a module as
a sort of extended handshake even w/ no body, but not against
heterogenous backends.

sf at sfritsch

Aug 3, 2013, 10:26 AM

Post #10 of 15 (72 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

Am Freitag, 2. August 2013, 11:21:56 schrieb Eric Covener:
> > I think this does not work for GET requests or request without a
> > request body.
> Just re-read spec, you are right -- we are abusing this in a module
> as a sort of extended handshake even w/ no body, but not against
> heterogenous backends.

One could do an 'OPTIONS *' request. But I am not sure if that is any
better than proxy-initial-not-pooled in terms of performance.

Or a filter could just send the first bytes of the request (less than
the request line) and then do a filter flush. If that fails, repeating
the request on a different connection would be ok even for non-
idempotent requests, because we would know that the backend has not
received the full request yet. I don't know how many errors this would
catch in practice, though.

jim at jaguNET

Aug 4, 2013, 8:16 AM

Post #11 of 15 (71 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

I've tried looking into that, and I found it more trouble
than it was worth... (I'm sure that the list archives have
posts about the 'http ping' tests). The problem is that the
OPTIONS request could be that request that kicks the backend
from being a keptalive connection to closing it. :/

On Aug 3, 2013, at 1:26 PM, Stefan Fritsch <sf [at] sfritsch> wrote:

> Am Freitag, 2. August 2013, 11:21:56 schrieb Eric Covener:
>>> I think this does not work for GET requests or request without a
>>> request body.
>> Just re-read spec, you are right -- we are abusing this in a module
>> as a sort of extended handshake even w/ no body, but not against
>> heterogenous backends.
>
> One could do an 'OPTIONS *' request. But I am not sure if that is any
> better than proxy-initial-not-pooled in terms of performance.
>
> Or a filter could just send the first bytes of the request (less than
> the request line) and then do a filter flush. If that fails, repeating
> the request on a different connection would be ok even for non-
> idempotent requests, because we would know that the backend has not
> received the full request yet. I don't know how many errors this would
> catch in practice, though.
>

thomas.r.w.eckert at gmail

Aug 4, 2013, 11:49 PM

Post #12 of 15 (57 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

> One could do an 'OPTIONS *' request. But I am not sure if that is any
better than proxy-initial-not-pooled in terms of performance.

I don't see why an OPTIONS request should not encounter problems where a
GET request will. After all, the problem is on the transport layer, not on
the application layer. Am I missing something ?

> Or a filter could just send the first bytes of the request (less than the
request line) and then do a filter flush. If that fails, repeating the
request on a different connection would be ok even for non- idempotent
requests, because we would know that the backend has not received the full
request yet. I don't know how many errors this would catch in practice,
though.

This is pretty much what I stated earlier. So I assume (re)opening a
connection and having the whole process of sending the request transition
to that (re)new(ed) connection is possible for an input filter. I was not
sure about it. Going to look into it once I have time..

On Sat, Aug 3, 2013 at 7:26 PM, Stefan Fritsch <sf [at] sfritsch> wrote:

> Am Freitag, 2. August 2013, 11:21:56 schrieb Eric Covener:
> > > I think this does not work for GET requests or request without a
> > > request body.
> > Just re-read spec, you are right -- we are abusing this in a module
> > as a sort of extended handshake even w/ no body, but not against
> > heterogenous backends.
>
> One could do an 'OPTIONS *' request. But I am not sure if that is any
> better than proxy-initial-not-pooled in terms of performance.
>
> Or a filter could just send the first bytes of the request (less than
> the request line) and then do a filter flush. If that fails, repeating
> the request on a different connection would be ok even for non-
> idempotent requests, because we would know that the backend has not
> received the full request yet. I don't know how many errors this would
> catch in practice, though.
>

covener at gmail

Aug 5, 2013, 4:18 AM

Post #13 of 15 (57 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert
<thomas.r.w.eckert [at] gmail> wrote:
>> One could do an 'OPTIONS *' request. But I am not sure if that is any
>> better than proxy-initial-not-pooled in terms of performance.
>
> I don't see why an OPTIONS request should not encounter problems where a GET
> request will. After all, the problem is on the transport layer, not on the
> application layer. Am I missing something ?

Could have problems at either level (e.g. MaxKeepalives)

rainer.jung at kippdata

Aug 5, 2013, 7:13 AM

Post #14 of 15 (57 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

On 05.08.2013 13:18, Eric Covener wrote:
> On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert
> <thomas.r.w.eckert [at] gmail> wrote:
>>> One could do an 'OPTIONS *' request. But I am not sure if that is any
>>> better than proxy-initial-not-pooled in terms of performance.
>>
>> I don't see why an OPTIONS request should not encounter problems where a GET
>> request will. After all, the problem is on the transport layer, not on the
>> application layer. Am I missing something ?
>
> Could have problems at either level (e.g. MaxKeepalives)

I think what people are trying to say: another request in front of each
request might increase the relative frequency (per real request) with
which the problem occurs.

Regards,

Rainer

jim at jaguNET

Aug 5, 2013, 8:07 AM

Post #15 of 15 (57 views)
Permalink

Re: mod_proxy, oooled backend connections and the keep-alive race condition [In reply to]

On Aug 5, 2013, at 10:13 AM, Rainer Jung <rainer.jung [at] kippdata> wrote:

> On 05.08.2013 13:18, Eric Covener wrote:
>> On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert
>> <thomas.r.w.eckert [at] gmail> wrote:
>>>> One could do an 'OPTIONS *' request. But I am not sure if that is any
>>>> better than proxy-initial-not-pooled in terms of performance.
>>>
>>> I don't see why an OPTIONS request should not encounter problems where a GET
>>> request will. After all, the problem is on the transport layer, not on the
>>> application layer. Am I missing something ?
>>
>> Could have problems at either level (e.g. MaxKeepalives)
>
> I think what people are trying to say: another request in front of each
> request might increase the relative frequency (per real request) with
> which the problem occurs.
>

+1

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads