Mailing List Archive: Apache: Dev

Event MPM

1 2

View All

Index | Next | Previous | View Threaded

chip at outoforder

Oct 24, 2004, 7:30 PM

Post #1 of 38 (2568 views)
Permalink

Event MPM

Several months ago, Greg Ames posted [1] his 'Event Driven MPM' patch as
a diff to the Worker MPM:
http://apache.org/~gregames/event.patch

I took this patch as the starting point for my 'Event MPM'.

My Patch is split into two parts.

First is a patch to APR that provides an extension to apr_pollset that
optionally make some parts of it threadsafe. This patch has been
submitted to APR before, and hopefully it will get accepted there soon.

The other patch is the actual MPM and with a few changes to other parts
of the source to take advantage of the Event Thread for handling Keep
Alive Requests.

These patches total over 160k. Since many mail servers are extremely
lame, I have put them up on the web at:
http://www.apache.org/~pquerna/event-mpm/

MPM Structure:
I have made the MPM a single process with multiple threads It currently
does not spawn new threads on demand, but I plan to add this soon. By
making it run as a single process, I was able to completely remove the
accept() locking. This also greatly simplifies the Listener Thread's
poll()`ing of the sockets.

The Listener thread includes all the Sockets that it is listening to for
incoming requests on in the same pollset as all requests waiting for
IO/KeepAlive. Greg's original patch had a separate 'Event' Thread, but
I have chosen to combine them. This can also be extended to include the
Lingering Close patch that Joe Schaefer submitted [2] this August.

The Worker Threads work much like the 'Worker' MPM, but when a
connection is ready for a Keep Alive, they will push the client back to
the Listener/Event Thread. This thread does not need to be woken up
like in Greg Ames' patch. This is because of the enhancement to
apr_pollset that enables other threads to _add() or _remove() while the
main thread is inside a _poll().

Super-Quick-Yet-Little-Real-Meaning-Benchmarks of Requests per second:

Non-KeepAlive:
`ab -c 25 -n 100000 http://10.10.10.10:6080/index.html`
Worker MPM: 2138.28
Event MPM: 2147.95

KeepAlive:
`ab -k -c 25 -n 100000 http://10.10.10.10:6080/index.html`
Worker MPM: 4396.38
Event MPM: 4119.40

(This is using the EPoll backend on a Linux 2.6 Machine, over 100mbit LAN.).

The place where the Event MPM should shine is with the more common case
of relatively high-latency Internet clients. The Event MPM isn't super
powerful on the KeepAlive-over-LAN case because it forces a context
switch to process the client again when it is does another request as
part of a KeepAlive.

I would like to get this into 2.1 CVS relatively soon. I plan to spend
more time cleaning it up and tuning it this week.

I know this email is light on some of the implementation details, feel
free to ask about anything I did :)

I would love feedback on all aspects of the patch. Please feel free to
rip it apart :)

-Paul Querna

[1] http://marc.theaimsgroup.com/?t=108670503400001&r=1&w=2
[2] http://marc.theaimsgroup.com/?t=109346431300004&r=1&w=2

bakins at web

Oct 25, 2004, 6:17 AM

Post #2 of 38 (2544 views)
Permalink

Re: Event MPM [In reply to]

Paul Querna wrote:

> Non-KeepAlive:
> `ab -c 25 -n 100000 http://10.10.10.10:6080/index.html`
> Worker MPM: 2138.28
> Event MPM: 2147.95
>
> KeepAlive:
> `ab -k -c 25 -n 100000 http://10.10.10.10:6080/index.html`
> Worker MPM: 4396.38
> Event MPM: 4119.40
>

Have you tried it with higher number of clients -- i.e,. -c 1024?

We are interesting in the event mpm mainly for dealing with keep alives.

--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies

chip at force-elite

Oct 25, 2004, 9:45 AM

Post #3 of 38 (2526 views)
Permalink

Re: Event MPM [In reply to]

Brian Akins wrote:
> Paul Querna wrote:
>
>> Non-KeepAlive:
>> `ab -c 25 -n 100000 http://10.10.10.10:6080/index.html`
>> Worker MPM: 2138.28
>> Event MPM: 2147.95
>>
>> KeepAlive:
>> `ab -k -c 25 -n 100000 http://10.10.10.10:6080/index.html`
>> Worker MPM: 4396.38
>> Event MPM: 4119.40
>>
>
> Have you tried it with higher number of clients -- i.e,. -c 1024?

Nope. I was already maxing out my 100mbit LAN at 25 clients. I don't
have a good testing area for static content request benchmarking.

I am thinking of trying to find an old pentium I with PCI and putting a
GigE card in it just for benchmarking.

The current patch does not spawn new threads on demand. You need to set
"ThreadsPerChild 1200" to test that many clients. This is on my short
list of things to change before committing the Event MPM to CVS.

> We are interesting in the event mpm mainly for dealing with keep alives.
>

Yes, this is the target the Event MPM aims at :)

-Paul Querna

ivanr at webkreator

Oct 25, 2004, 10:28 AM

Post #4 of 38 (2564 views)
Permalink

Re: Event MPM [In reply to]

Paul Querna wrote:
>
> Brian Akins wrote:
>
>> We are interesting in the event mpm mainly for dealing with keep alives.
>>
> Yes, this is the target the Event MPM aims at :)

If I understand the nature of the patch correctly then you don't
need to go increasing the number of clients at all. Instead you
should be looking at changing ab to pause for a few seconds
between every two requests carried out over the same connection.
With that change, and limiting the maximum number of processes to,
say, 100, you should be able to obtain meaningful results with your
current setup. Worker MPM will max out at some point, with Event MPM
going past it to max at a higher value.

--
ModSecurity (http://www.modsecurity.org)
[ Open source IDS for Web applications ]

brianp at apache

Oct 25, 2004, 10:34 AM

Post #5 of 38 (2556 views)
Permalink

Re: Event MPM [In reply to]

Paul Querna wrote:

> Brian Akins wrote:
>
>> Paul Querna wrote:
>
[...]

>> Have you tried it with higher number of clients -- i.e,. -c 1024?
>
>
> Nope. I was already maxing out my 100mbit LAN at 25 clients. I don't
> have a good testing area for static content request benchmarking.
>
> I am thinking of trying to find an old pentium I with PCI and putting
> a GigE card in it just for benchmarking.

How about modifying ab to add a delay of a second or two between
successive read(2) calls
on the same connection (and limiting the max read size to a small value,
to make sure each
response requires multiple reads)? The throughput numbers wouldn't be
very impressive, of
course, but you'd be able to see how much memory and how many threads
the Event MPM
uses in handling a real-world number of concurrent connections, compared
to Worker.

> The current patch does not spawn new threads on demand. You need to
> set "ThreadsPerChild 1200" to test that many clients. This is on my
> short list of things to change before committing the Event MPM to CVS.

This part seems surprising. What's the relationship between clients and
threads in the
Event MPM--does it use a thread per connection?

I haven't had a chance to check out the Event MPM yet, but I'm planning
to download it and
study the code later this week.

For what it's worth, I can think of at least one killer app for the
Event MPM beyond keep-alive
handling: as an efficient (low memory & thread use per connection)
connection multiplexer in
front of an inefficient (high memory & thread use per connection) appserver.

-Brian

gregames at remulak

Oct 25, 2004, 10:41 AM

Post #6 of 38 (2559 views)
Permalink

Re: Event MPM [In reply to]

++1 for moving this along.

For any newbies, the object of the game is to increase httpd scalability by
reducing the number of worker threads and their associated stack memory.

Paul Querna wrote:

> First is a patch to APR that provides an extension to apr_pollset that
> optionally make some parts of it threadsafe. This patch has been
> submitted to APR before, and hopefully it will get accepted there soon.

I need to catch up here. quick questions: will this event MPM work with
conventional poll (i.e. no epoll or kqueue)? Can it handle a worker thread
adding directly to the pollset?

> The other patch is the actual MPM and with a few changes to other parts
> of the source to take advantage of the Event Thread for handling Keep
> Alive Requests.

IMO the "few changes to other parts" are the most important to do first. If we
can get consensus on the core parts and commit them to 2.1, much of the work can
be done in the experimental mpm dir without bothering uninterested parties.

> These patches total over 160k. Since many mail servers are extremely
> lame, I have put them up on the web at:
> http://www.apache.org/~pquerna/event-mpm/

diffing worker.c to event.c and worker/fdqueue.c to event/fdqueue.c should make
the mpm changes easier to follow.

> MPM Structure:
> I have made the MPM a single process with multiple threads

interesting! but then per process limits (fd's esp. for sockets, virtual
memory) could put a ceiling on how high this can scale. Plus if a buggy module
seg faults, it's a lot more disruptive when there's only one worker process.

> It currently
> does not spawn new threads on demand, but I plan to add this soon. By
> making it run as a single process, I was able to completely remove the
> accept() locking. This also greatly simplifies the Listener Thread's
> poll()`ing of the sockets.

understood, but see above. My vote is to keep it multiprocess capable and
perhaps allow a single process ./configure option.

> The Listener thread includes all the Sockets that it is listening to for
> incoming requests on in the same pollset as all requests waiting for
> IO/KeepAlive. Greg's original patch had a separate 'Event' Thread, but
> I have chosen to combine them.

+1

That means that some of the changes to how fdqueue.c manipulates "idlers" aren't
strictly necessary since the listener is the only thread that ever tries to
reserve a worker thread. OTOH I don't think those changes to fdqueue.c hurt
anything either so it's a minor detail.

> The Worker Threads work much like the 'Worker' MPM, but when a
> connection is ready for a Keep Alive, they will push the client back to
> the Listener/Event Thread. This thread does not need to be woken up
> like in Greg Ames' patch. This is because of the enhancement to
> apr_pollset that enables other threads to _add() or _remove() while the
> main thread is inside a _poll().

even with plain ol' vanilla poll() ?

> The place where the Event MPM should shine is with the more common case
> of relatively high-latency Internet clients. The Event MPM isn't super
> powerful on the KeepAlive-over-LAN case because it forces a context
> switch to process the client again when it is does another request as
> part of a KeepAlive.

I heard of an application where many clients send a heartbeat to a server over
http every few seconds. That would benefit greatly by using keepalives and some
form of event MPM, even over a LAN.

Greg

gregames at remulak

Oct 25, 2004, 11:17 AM

Post #7 of 38 (2543 views)
Permalink

Re: Event MPM [In reply to]

Brian Pane wrote:
> Paul Querna wrote:
>
>> Brian Akins wrote:

>>> Have you tried it with higher number of clients -- i.e,. -c 1024?

>> Nope. I was already maxing out my 100mbit LAN at 25 clients. I don't
>> have a good testing area for static content request benchmarking.

>> I am thinking of trying to find an old pentium I with PCI and putting
>> a GigE card in it just for benchmarking.

I'd love to find one like this too. I sometimes use a 180 MHz Pentium Pro box.
But it only has 160M of EDO memory which is hard to scrounge around here, and
probably can't use modern hard drives effectively due to BIOS limitations.

> How about modifying ab to add a delay of a second or two between
> successive read(2) calls
> on the same connection

an excellent idea. Or maybe add a delay after any read if -k is in effect.

>> The current patch does not spawn new threads on demand. You need to
>> set "ThreadsPerChild 1200" to test that many clients. This is on my
>> short list of things to change before committing the Event MPM to CVS.

> This part seems surprising. What's the relationship between clients and
> threads in the
> Event MPM--does it use a thread per connection?

one thread per connection with an active http request, plus the listener/event
thread who owns all the connections in keepalive. I believe Paul is saying set
ThreadsPerChild to 1200 to handle the worst case behavior - 100% of the
connections are doing real work at some instant and none are in keepalive
timeouts.

Greg

chip at force-elite

Oct 25, 2004, 11:25 AM

Post #8 of 38 (2561 views)
Permalink

Re: Event MPM [In reply to]

Greg Ames wrote:
>> First is a patch to APR that provides an extension to apr_pollset that
>> optionally make some parts of it threadsafe. This patch has been
>> submitted to APR before, and hopefully it will get accepted there soon.
>
>
> I need to catch up here. quick questions: will this event MPM work with
> conventional poll (i.e. no epoll or kqueue)? Can it handle a worker
> thread adding directly to the pollset?

The patch to APR adds a new 'APR_POLLSET_THREADSAFE' flag for
apr_pollset_create(...).

This only works for the EPoll and KQueue backends. This allows a worker
thread to _add() or _remove() directly, without having to touch the
thread in _poll().

It could be implemented for plain Poll by having the pollset contain an
internal pipe. This Pipe could be pushed by an _add() or _remove() and
force the _poll() thread to wake up. The thread in _poll() could then
add or remove any sockets from it's set, and then start the _poll()
again.....

This is how your original patch did it, but we could push it down to
APR. I don't think in the end it will yield good performance with this
method.

My personal view is that most platforms will have either EPoll() or
KQueue() Support. I guess its only an issue for people running 2.2./2.4
Linux, since KQueue has been in *BSD for a long time. As time passes, I
believe that implementing it for plain poll() will be come less important.

Other platforms like Solaris have better poll() replacements that we
should add to apr_pollset_*.

[...snip...]
>> I have made the MPM a single process with multiple threads
>
> interesting! but then per process limits (fd's esp. for sockets,
> virtual memory) could put a ceiling on how high this can scale. Plus if
> a buggy module seg faults, it's a lot more disruptive when there's only
> one worker process.

We don't have buggy modules :-)

Isn't the per-process FD limit only really a problem for platforms like
older Solaris? I thought that on Linux/BSD it is quite high by default now.

>> It currently does not spawn new threads on demand, but I plan to
>> add this soon. By making it run as a single process, I was able to
>> completely remove the accept() locking. This also greatly simplifies
>> the Listener Thread's poll()`ing of the sockets.
>
>
> understood, but see above. My vote is to keep it multiprocess capable
> and perhaps allow a single process ./configure option.

I am not completely adverse to putting multi-processes back in, I mostly
wanted to get a working patch out there for review.

The current logic used in Worker is flawed for poll`ing of Listening
sockets. I just ripped all of it out, and ended up with a much simpler
MPM that avoids extra locking.

[...snip...]
>> The Worker Threads work much like the 'Worker' MPM, but when a
>> connection is ready for a Keep Alive, they will push the client back
>> to the Listener/Event Thread. This thread does not need to be woken
>> up like in Greg Ames' patch. This is because of the enhancement to
>> apr_pollset that enables other threads to _add() or _remove() while
>> the main thread is inside a _poll().
>
>
> even with plain ol' vanilla poll() ?
>

Nope. Its just not easily to implement and support plain old vanilla
poll(). You cannot just add an FD to its array of sockets from other
threads. The poll() will also not see any changes in its list until you
wake it up, and start the poll() again.

This is not impossible to implement, but its going to have a huge number
of context switches for what should be a very common operation.

[...snip..]
>> The place where the Event MPM should shine is with the more common
>> case of relatively high-latency Internet clients. The Event MPM isn't
>> super powerful on the KeepAlive-over-LAN case because it forces a
>> context switch to process the client again when it is does another
>> request as part of a KeepAlive.
>
>
> I heard of an application where many clients send a heartbeat to a
> server over http every few seconds. That would benefit greatly by using
> keepalives and some form of event MPM, even over a LAN.

Yup.

Another place the event MPM would rock is on an FTP server. (see mod_ftpd).

Yet Another place is an IMAPv4 server written as an Apache Module. (This
is on my ~/TODO list)

Thanks for the Feedback!

-Paul Querna

chip at force-elite

Oct 25, 2004, 11:45 AM

Post #9 of 38 (2554 views)
Permalink

Re: Event MPM [In reply to]

Brian Pane wrote:
> How about modifying ab to add a delay of a second or two between
> successive read(2) calls
> on the same connection (and limiting the max read size to a small value,
> to make sure each
> response requires multiple reads)? The throughput numbers wouldn't be
> very impressive, of
> course, but you'd be able to see how much memory and how many threads
> the Event MPM
> uses in handling a real-world number of concurrent connections, compared
> to Worker.

+1, I will look at patching this tonight. I think some other HTTP
benchmarkers might already have this feature. Flood?

>> The current patch does not spawn new threads on demand. You need to
>> set "ThreadsPerChild 1200" to test that many clients. This is on my
>> short list of things to change before committing the Event MPM to CVS.
>
> This part seems surprising. What's the relationship between clients and
> threads in the
> Event MPM--does it use a thread per connection?

A thread per-connection that is currently being processed.

Note that this is not the traditional 'event' model that people write
huge papers about and thttpd raves about, but rather a hybrid that uses
a Worker Thread todo the processing, and a single 'event' thread to
handle places where we are waiting for IO. (Currently accept() and Keep
Alive Requests). Perhaps it needs a different name? (eworker?)

A future direction to investigate would be to make all of the initial
Header parsing be done Async, and then switch to a Worker thread to
preform all the post_read hooks, handlers, and filters. I believe this
could be done without breaking many 3rd party modules. (SSL could be a
bigger challenge here...)

> I haven't had a chance to check out the Event MPM yet, but I'm planning
> to download it and
> study the code later this week.
>
> For what it's worth, I can think of at least one killer app for the
> Event MPM beyond keep-alive
> handling: as an efficient (low memory & thread use per connection)
> connection multiplexer in
> front of an inefficient (high memory & thread use per connection)
> appserver.

Yes, another thing on my ~/TODO is to use this design this as the base
for a perchild replacement.

My idea is to use a lightweight 'event' frontend to determine which
backend to pass a socket to. These backends would be running as
different UIDs, and could be running a threaded or non-threaded manner
to optionally support PHP.

-Paul Querna

chip at force-elite

Oct 25, 2004, 11:53 AM

Post #10 of 38 (2540 views)
Permalink

Re: Event MPM [In reply to]

Paul Querna wrote:
> A thread per-connection that is currently being processed.
>
> Note that this is not the traditional 'event' model that people write
> huge papers about and thttpd raves about, but rather a hybrid that uses
> a Worker Thread todo the processing, and a single 'event' thread to
> handle places where we are waiting for IO. (Currently accept() and Keep
> Alive Requests). Perhaps it needs a different name? (eworker?)
>
> A future direction to investigate would be to make all of the initial
> Header parsing be done Async, and then switch to a Worker thread to
> preform all the post_read hooks, handlers, and filters. I believe this
> could be done without breaking many 3rd party modules. (SSL could be a
> bigger challenge here...)

Some academics have played with this model of a event thread + worker
threads. Best I can find is the Java 'SEDA: An Architecture for Highly
Concurrent Server Applications'. [1]

They have some pretty graphs in their papers of their Haboob web server
that can scale very nicely compared to traditional event or
pure-threaded servers.[2] [3]

-Paul Querna

[1] http://www.eecs.harvard.edu/~mdw/proj/seda/
[2] http://www.enhyper.com/content/eventsbadidea.pdf
[3] http://www.cis.upenn.edu/~hhl/cse434/lectures/seda.pdf

bakins at web

Oct 25, 2004, 12:06 PM

Post #11 of 38 (2534 views)
Permalink

Re: Event MPM [In reply to]

Greg Ames wrote:

> one thread per connection with an active http request, plus the
> listener/event thread who owns all the connections in keepalive. I
> believe Paul is saying set ThreadsPerChild to 1200 to handle the worst
> case behavior - 100% of the connections are doing real work at some
> instant and none are in keepalive timeouts.

Can you still have multiple processes? We use 10k plus threads per box
with worker.

--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies

gregames at remulak

Oct 25, 2004, 12:37 PM

Post #12 of 38 (2521 views)
Permalink

Re: Event MPM [In reply to]

Brian Akins wrote:

> Can you still have multiple processes? We use 10k plus threads per box
> with worker.

with my patch, yes. with Paul's, no.

But Paul's has some very nice features that mine doesn't have, so I think a
hybrid is the way to go.

Assuming you have a high percentage of threads in keepalive timeouts, you will
be able to cut down on the number of threads per box.

Greg

bakins at web

Oct 25, 2004, 12:54 PM

Post #13 of 38 (2535 views)
Permalink

Re: Event MPM [In reply to]

Greg Ames wrote:

> Assuming you have a high percentage of threads in keepalive timeouts,
> you will be able to cut down on the number of threads per box.

Yes, we do.

I can certainly provide you guys with some testing, if nothing else. We
have some home grown benchmarks that may could help, not to mention lots
of live traffic :)

--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies

gregames at remulak

Oct 25, 2004, 1:10 PM

Post #14 of 38 (2541 views)
Permalink

Re: Event MPM [In reply to]

Brian Akins wrote:

> I can certainly provide you guys with some testing, if nothing else.

excellent!

> We have some home grown benchmarks that may could help,

If they simulate user think time or would otherwise cause a lot of keepalive
timeouts, great! Finding the right client/benchmark is a problem for me right
now and I believe for Paul too.

> not to mention lots of live traffic :)

ummm, I'm glad you put a smiley on the end of that :)

Greg

chip at force-elite

Oct 25, 2004, 1:28 PM

Post #15 of 38 (2556 views)
Permalink

Re: Event MPM [In reply to]

Greg Ames wrote:
> Brian Akins wrote:
>
>> Can you still have multiple processes? We use 10k plus threads per
>> box with worker.
>
>
> with my patch, yes. with Paul's, no.
>

Correct at the moment.

I think based on the feedback so far, I will investigate making it
multi-processed again.

chip at force-elite

Oct 25, 2004, 1:32 PM

Post #16 of 38 (2519 views)
Permalink

Re: Event MPM [In reply to]

Greg Ames wrote:
> Brian Akins wrote:
>> We have some home grown benchmarks that may could help,
>
>
> If they simulate user think time or would otherwise cause a lot of
> keepalive timeouts, great! Finding the right client/benchmark is a
> problem for me right now and I believe for Paul too.

Yup. This is my biggest problem right now with the benchmarks.

`ab` just doesn't give a realistic modeling of how clients use Keep
Alives. If you have some home grown programs that do, I would love to
use them.

gregames at remulak

Oct 25, 2004, 2:45 PM

Post #17 of 38 (2525 views)
Permalink

Re: Event MPM [In reply to]

Paul Querna wrote:

> This only works for the EPoll and KQueue backends. This allows a worker
> thread to _add() or _remove() directly, without having to touch the
> thread in _poll().
>
> It could be implemented for plain Poll by having the pollset contain an
> internal pipe. This Pipe could be pushed by an _add() or _remove() and
> force the _poll() thread to wake up. The thread in _poll() could then
> add or remove any sockets from it's set, and then start the _poll()
> again.....
>
> This is how your original patch did it, but we could push it down to
> APR.

let's stick with your version at least for now.

> I don't think in the end it will yield good performance with this method.

agreed. I wasn't happy with the extra pipe and the overhead, though some of
that could go away. The point was to get something out there as a
proof-of-concept that would work for just about anybody. Today the barrier to
entry is much lower in the Linux world anyway. I'm not sure if a distro with a
2.6 kernel will install easily on my Pentium Pro benchmarking box, but that's a
pretty weird environment.

Greg

justin at erenkrantz

Oct 25, 2004, 10:23 PM

Post #18 of 38 (2528 views)
Permalink

Re: Event MPM [In reply to]

--On Monday, October 25, 2004 2:17 PM -0400 Greg Ames <gregames [at] remulak>
wrote:

>>> I am thinking of trying to find an old pentium I with PCI and putting
>>> a GigE card in it just for benchmarking.
>
> I'd love to find one like this too. I sometimes use a 180 MHz Pentium Pro
> box. But it only has 160M of EDO memory which is hard to scrounge around
> here, and probably can't use modern hard drives effectively due to BIOS
> limitations.

Good luck. I really wouldn't recommending throwing money down this path. I
spent $300 I really don't have on a GigE switch and cards in order to get more
out of mod_cache to little avail other than more useless benchmark numbers.

Yet, if you send me some flood scripts and httpd patches, I can throw some
numbers back at ya though. No sense other httpd'ers wasting their money.

>> How about modifying ab to add a delay of a second or two between
>> successive read(2) calls
>> on the same connection
>
> an excellent idea. Or maybe add a delay after any read if -k is in effect.

Um, flood already lets you do this and lots more. ab isn't a load simulator
and trying to produce benchmarks with it is laughable. -- justin

justin at erenkrantz

Oct 25, 2004, 11:59 PM

Post #19 of 38 (2569 views)
Permalink

Re: Event MPM [In reply to]

--On Sunday, October 24, 2004 8:30 PM -0600 Paul Querna <chip [at] outoforder>
wrote:

> I took this patch as the starting point for my 'Event MPM'.
...
> I would love feedback on all aspects of the patch. Please feel free to rip
> it apart :)

I just had a conversation on #apr with Paul. But, I'll rehash the one issue I
brought up there: this MPM breaks any pipelined connections because there can
be a deadlock.

core_input_filter or any connection-level filter (say SSL) could be holding
onto a complete request that hasn't been processed yet. The worker thread
will only process one request and then put it back on the stack. But, there's
certainly no reason why another request isn't already in the chain ready to be
read. And, the listener/event thread will be waiting for more data to come in
- but, we already read it. Oops. (And, perhaps, it's not enough to be a
complete request - so it'd block defeating the purpose of the event thread -
Oops again.)

I'll note that serf (the async HTTP client library Greg and I are working on)
had to deal with this same case. The only suggestion I have is that
ap_read_request() needs to be come async/non-blocking so that it can pick up
requests that aren't 'ready' yet and return an 'EAGAIN' status code. This is
what we do with serf, but httpd doesn't have that luxury.

Greg, Manoj, and I (and many others) have talked for a long time about having
a fully async MPM - on both read *and* writes we hand the thread over to the
event thread. When there is data to be read or we can write, we assign it to
a thread. This MPM is a step in the right direction towards that goal, but
we'd have to solve the 'unprocessed, but read request' problem first.

FWIW, if we start to go down this route, to me this smells like 2.3 candidate
work. This is likely going to snowball real fast into other areas and I'd
really like to keep us close to seriously discussing 2.2 at AC in a few weeks
instead of throwing HEAD into turmoil with these changes.

I'll also note that flood would almost certainly trigger this as it uses
pipelined requests - unlike ab which doesn't know pipelining. =) -- justin

gregames at remulak

Oct 26, 2004, 8:24 AM

Post #20 of 38 (2503 views)
Permalink

Re: Event MPM [In reply to]

Justin Erenkrantz wrote:

> Um, flood already lets you do this and lots more.

Does flood allow multiple connections per thread or per process? Ideally the
load simulator would scale as well as the server, although that's not strictly
necessary. Eventually I want to run at least as many connections as Colm has -
20k last I heard.

Greg

dnicklay at web

Oct 26, 2004, 8:32 AM

Post #21 of 38 (2558 views)
Permalink

Re: Event MPM [In reply to]

Hi,

>>> How about modifying ab to add a delay of a second or two between
>>> successive read(2) calls
>>> on the same connection

okay

>
> Um, flood already lets you do this and lots more. ab isn't a load
> simulator and trying to produce benchmarks with it is laughable. -- justin
>

ab is very good at pounding very hard at servers, and it can be used for
benchmarks with some tweaking. It is not at all laughable that there
are not that many benchmarkers that can handle the gigabit data flow
combined with > 10K simultaneous connections. We have tried MANY of
them, and have found all of them wanting in some respect or another.

I put together a modified ab that can simulate any connection speed,
though I still have to add in the code to pause for intervals between
keepalives. I also hacked out as much of the dependency on a single URL
as I could. Source is attached.

Note: this is a work in progress, so it is still pretty ugly. I also
have a long list of things that still need to be added to make it
simulate real life traffic. But perhaps someone else can help me work
on it?

--
David Nicklay O-
Location: CNN Center - SE0811A
Office: 404-827-2698 Cell: 404-545-6218

Attachments:

apbm.c (68.5 KB)

justin at erenkrantz

Oct 26, 2004, 8:34 AM

Post #22 of 38 (2524 views)
Permalink

Re: Event MPM [In reply to]

--On Tuesday, October 26, 2004 11:24 AM -0400 Greg Ames <gregames [at] remulak>
wrote:

> Does flood allow multiple connections per thread or per process? Ideally
> the load simulator would scale as well as the server, although that's not
> strictly necessary. Eventually I want to run at least as many connections
> as Colm has - 20k last I heard.

The threads are fairly lightweight. But, remember, that you can run flood on
multiple machines. ;-) -- justin

gregames at remulak

Oct 26, 2004, 9:03 AM

Post #23 of 38 (2563 views)
Permalink

Re: Event MPM [In reply to]

Justin Erenkrantz wrote:

> this MPM breaks any pipelined connections because there can be a deadlock.
>
> core_input_filter or any connection-level filter (say SSL) could be
> holding onto a complete request that hasn't been processed yet. The
> worker thread will only process one request and then put it back on the
> stack. But, there's certainly no reason why another request isn't
> already in the chain ready to be read. And, the listener/event thread
> will be waiting for more data to come in - but, we already read it.
> Oops.

Yes, this needs to be fixed. I don't see it as difficult problem. We already
test for whether the output filters need to be flushed (i.e., is there any more
data in the input filter chain) before the request ends. We just need need to
remember the outcome of that test and react appropriately.

Are there a lot of browsers out there that implement true pipelining? I know
it's in the RFC for good reasons but I don't believe I've ever seen it in the wild.

(And, perhaps, it's not enough to be a complete request - so it'd
> block defeating the purpose of the event thread

umm, I disagree. I'm not too concerned about worker threads blocking in socket
reads occasionally. The point here is to go after the low hanging fruit
(frequent long delays between requests) without massive code churn in the server.

Greg

justin at erenkrantz

Oct 26, 2004, 9:31 AM

Post #24 of 38 (2558 views)
Permalink

Re: Event MPM [In reply to]

--On Tuesday, October 26, 2004 12:03 PM -0400 Greg Ames <gregames [at] remulak>
wrote:

> Yes, this needs to be fixed. I don't see it as difficult problem. We
> already test for whether the output filters need to be flushed (i.e., is
> there any more data in the input filter chain) before the request ends. We
> just need need to remember the outcome of that test and react appropriately.

This isn't about whether the output filters need to be flushed: this is about
whether the connection-level input filters have already read data from the
socket and are hanging onto data that needs to be processed before the client
will send more data. The event MPM will be sitting on a poll() waiting for
the client to send more data when the client already has sent the next request.

> Are there a lot of browsers out there that implement true pipelining? I
> know it's in the RFC for good reasons but I don't believe I've ever seen it
> in the wild.

Mozilla, Opera, Squid, etc. I'm, uh, absolutely against any code being
checked in that breaks HTTP pipelining. (You should get the drift of where
I'm going.)

> umm, I disagree. I'm not too concerned about worker threads blocking in
> socket reads occasionally. The point here is to go after the low hanging
> fruit (frequent long delays between requests) without massive code churn in
> the server.

Certainly not at the expense of HTTP/1.1-compliance. -- justin

gregames at remulak

Oct 26, 2004, 10:53 AM

Post #25 of 38 (2557 views)
Permalink

Re: Event MPM [In reply to]

Justin Erenkrantz wrote:
> --On Tuesday, October 26, 2004 12:03 PM -0400 Greg Ames
> <gregames [at] remulak> wrote:
>
>> Yes, this needs to be fixed. I don't see it as difficult problem. We
>> already test for whether the output filters need to be flushed (i.e., is
>> there any more data in the input filter chain) before the request
>> ends. We
>> just need need to remember the outcome of that test and react
>> appropriately.
>
>
> This isn't about whether the output filters need to be flushed: this is
> about whether the connection-level input filters have already read data
> from the socket and are hanging onto data that needs to be processed
> before the client will send more data.

right, I understand. check_pipeline_flush already tests whether the input
filters hold any data if I'm not mistaken.

> The event MPM will be sitting on
> a poll() waiting for the client to send more data when the client
> already has sent the next request.

not if we modify it so that the worker thread doesn't give up the connection to
the event thread when there is more data in the input filters. That's what I
meant by "react appropriately". Sorry if I wasn't clear.

Greg

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads