Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Apache: Dev

AsyncRequestWorkerFactor / Event Sizing / Closing Connections

 

 

Apache dev RSS feed   Index | Next | Previous | View Threaded


rainer.jung at kippdata

Feb 21, 2012, 7:53 PM

Post #1 of 5 (436 views)
Permalink
AsyncRequestWorkerFactor / Event Sizing / Closing Connections

Looking at the server status on www.apache.org running 2.3.15 one can
see, that about 50% of the async connections are in closing state.

We created AsyncRequestWorkerFactor to control the amount of
overcommitment in terms of connections relative to idle workers we allow
for each process. The formula is

Connections =

ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =

busy_workers + ((AsyncRequestWorkerFactor+1) * number of idle workers)

Default value for AsyncRequestWorkerFactor is "2", so per default

Connections = busy + 3 * idle.

Now if the situation on www.apache.org turns out to be typical, 50% of
Connections are in closing state, but we don't expect those to need any
more worker thread. So the overcommitment would reduce from a factor of
3 to something closer to 1.5, which isn't much.

Would it be feasible to only count the number of non-closing
connections, so allow new connections for a process as long as

Non_Closing_Connections <=

ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =

Comments?

A related question: again on www.apache.org not only the number of async
connections in closing state is pretty high, also the percentage of sync
connections in state "C" is about 50%! I don't really understand that,
usually (no big event experience yet) I see "C" only occasionally, not
with such a big percentage. E.g. on the EU mirror running 2.2 and having
less traffic there is hardly ever a "C" to notice. Is that an expected
2.4/event difference, or is there something strange with the us network
infrastructure?

Regards,

Rainer


d.s at daniel

Feb 22, 2012, 12:53 AM

Post #2 of 5 (422 views)
Permalink
Re: AsyncRequestWorkerFactor / Event Sizing / Closing Connections [In reply to]

Rainer Jung wrote on Wed, Feb 22, 2012 at 04:53:34 +0100:
> "C" only occasionally, not with such a big percentage. E.g. on the
> EU mirror running 2.2 and having less traffic there is hardly ever a
> "C" to notice. Is that an expected 2.4/event difference, or is there
> something strange with the us network infrastructure?

The EU mirror is not in the geo DNS rotation at this time. The only
traffic it sees it are people who type foo.eu.apache.org directly into
their browser.

(which probably happens occasionally for www.eu and never for the other
vhosts)


rainer.jung at kippdata

Feb 22, 2012, 3:48 AM

Post #3 of 5 (432 views)
Permalink
Re: AsyncRequestWorkerFactor / Event Sizing / Closing Connections [In reply to]

On 22.02.2012 09:53, Daniel Shahaf wrote:
> Rainer Jung wrote on Wed, Feb 22, 2012 at 04:53:34 +0100:
>> "C" only occasionally, not with such a big percentage. E.g. on the
>> EU mirror running 2.2 and having less traffic there is hardly ever a
>> "C" to notice. Is that an expected 2.4/event difference, or is there
>> something strange with the us network infrastructure?
>
> The EU mirror is not in the geo DNS rotation at this time. The only
> traffic it sees it are people who type foo.eu.apache.org directly into
> their browser.
>
> (which probably happens occasionally for www.eu and never for the other
> vhosts)

Yes but still 50% is quite high. Code says "C" is also during lingeirng
close. We would also start lingring close if there would be no idle
worker available, so it could also indicate a problem.

Nevertheless I still think the closing sockets shouldn't count against
AsyncRequestWorkerFactor.

Regards,

Rainer


sf at sfritsch

Feb 22, 2012, 1:34 PM

Post #4 of 5 (425 views)
Permalink
Re: AsyncRequestWorkerFactor / Event Sizing / Closing Connections [In reply to]

On Wed, 22 Feb 2012, Rainer Jung wrote:

> Looking at the server status on www.apache.org running 2.3.15 one can see,
> that about 50% of the async connections are in closing state.
>
> We created AsyncRequestWorkerFactor to control the amount of overcommitment
> in terms of connections relative to idle workers we allow for each process.
> The formula is
>
> Connections =
>
> ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =
>
> busy_workers + ((AsyncRequestWorkerFactor+1) * number of idle workers)
>
> Default value for AsyncRequestWorkerFactor is "2", so per default
>
> Connections = busy + 3 * idle.
>
> Now if the situation on www.apache.org turns out to be typical, 50% of
> Connections are in closing state, but we don't expect those to need any more
> worker thread. So the overcommitment would reduce from a factor of 3 to
> something closer to 1.5, which isn't much.
>
> Would it be feasible to only count the number of non-closing connections, so
> allow new connections for a process as long as
>
> Non_Closing_Connections <=
>
> ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =
>
> Comments?

That's certainly a possible improvement. Others are also possible. For
example, it should be possible to count ssl connections separately and
don't take them into account when calculating overcommitment.

Or currently, after a timeout during write completion, there is another
wait time for a lingering close. I don't think we need the full 30s wait
time here.

But there is also a bug that ap_start_lingering_close() is called from the
listener thread but may block (PR 52229). Maybe this should be fixed first
before doing more optimizations?

> A related question: again on www.apache.org not only the number of async
> connections in closing state is pretty high, also the percentage of sync
> connections in state "C" is about 50%! I don't really understand that,
> usually (no big event experience yet) I see "C" only occasionally, not with
> such a big percentage. E.g. on the EU mirror running 2.2 and having less
> traffic there is hardly ever a "C" to notice. Is that an expected 2.4/event
> difference, or is there something strange with the us network infrastructure?

Does www.apache.org do a lot of https? Maybe the "C" connections come from
SSL? Also, cgi requests where the response does not fit into the write
buffer are done synchronously since the recent change. Though probably
those should appear as "W".

BTW, mod_status should show if a slot is http or https. That's IMHO more
important thatn the HTTP version or even the method. And it would be nice
if mpm event could display more statistics in mod_status.


sf at sfritsch

Mar 2, 2012, 12:18 PM

Post #5 of 5 (400 views)
Permalink
Re: AsyncRequestWorkerFactor / Event Sizing / Closing Connections [In reply to]

On Wednesday 22 February 2012, Stefan Fritsch wrote:
> On Wed, 22 Feb 2012, Rainer Jung wrote:
> > Looking at the server status on www.apache.org running 2.3.15 one
> > can see, that about 50% of the async connections are in closing
> > state.
> >
> > We created AsyncRequestWorkerFactor to control the amount of
> > overcommitment in terms of connections relative to idle workers
> > we allow for each process. The formula is
> >
> > Connections =
> >
> > ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =
> >
> > busy_workers + ((AsyncRequestWorkerFactor+1) * number of idle
> > workers)
> >
> > Default value for AsyncRequestWorkerFactor is "2", so per default
> >
> > Connections = busy + 3 * idle.
> >
> > Now if the situation on www.apache.org turns out to be typical,
> > 50% of Connections are in closing state, but we don't expect
> > those to need any more worker thread. So the overcommitment
> > would reduce from a factor of 3 to something closer to 1.5,
> > which isn't much.
> >
> > Would it be feasible to only count the number of non-closing
> > connections, so allow new connections for a process as long as
> >
> > Non_Closing_Connections <=
> >
> > ThreadsPerChild + (AsyncRequestWorkerFactor * idle_workers) =
> >
> > Comments?
>
> That's certainly a possible improvement.

I did the attached patches last weekend. They should do it, but I
haven't had any chance to test them properly and won't have time in
the next 1-2 weeks. If someone wants to try them, please go ahead.

Cheers,
Stefan
Attachments: 0002-don-t-count-lingering-conns-agains-asyncworkerfactor.patch (2.30 KB)
  0001-add-counters-for-clogged-lingering-and-suspended-con.patch (6.18 KB)

Apache dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.