Mailing List Archive: Apache: Dev

Crazy slowloris mitigation patch

Index | Next | Previous | View Threaded

bojan at rexursive

Oct 15, 2009, 2:00 PM

Post #1 of 21 (2192 views)
Permalink

Crazy slowloris mitigation patch

While playing with slowloris against prefork, I wrote the attached
craziness.

I had httpd under slowloris attack (which would normally completely DOS
the server) and it seems that the attached patch made it handle the
requests. Sure, there was a lot of carnage in the process (child
processes dying), but somehow it made it better.

Laugh away... :-)

--
Bojan

Attachments:

httpd-kill_busy_read.patch (1.28 KB)

bojan at rexursive

Oct 15, 2009, 5:20 PM

Post #2 of 21 (2106 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Fri, 2009-10-16 at 08:00 +1100, Bojan Smojver wrote:

> + ap_mpm_safe_kill(reader, SIGKILL);

Actually, this can be SIGTERM too. Still does the job.

--
Bojan

bojan at rexursive

Oct 15, 2009, 6:31 PM

Post #3 of 21 (2093 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Fri, 2009-10-16 at 08:00 +1100, Bojan Smojver wrote:
> I wrote the attached craziness.

Slightly more sophisticated craziness attached.

--
Bojan

Attachments:

httpd-kill_busy_read.patch (2.37 KB)

bojan at rexursive

Oct 15, 2009, 8:36 PM

Post #4 of 21 (2097 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Fri, 2009-10-16 at 12:31 +1100, Bojan Smojver wrote:
> Slightly more sophisticated craziness attached.

OK, just a little bit cleaner this time.

--
Bojan

Attachments:

httpd-kill_busy_read.patch (2.52 KB)

bojan at rexursive

Oct 18, 2009, 12:35 AM

Post #5 of 21 (2075 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Fri, 2009-10-16 at 08:00 +1100, Bojan Smojver wrote:
> While playing with slowloris against prefork, I wrote the attached
> craziness.

Here is another take on the problem, call it "Craziness 2.0".

The idea here is that a busy server is highly unlikely to be stuck
reading using all its children over a maintenance interval (i.e. we
expect at least one of those readers to turn into something else during
the interval).

So, if we find that all children are stuck reading, we simply close the
sockets (SIGINT was chosen arbitrarily for this - it should probably be
something else). After closing the socket by force, we go back to
handling new requests, but this time without the need to fork more
children, which should be more gentle on the machine running the thing.

Similar approach may work for worker too, but sockets would have to be
remembered in thread specific variables and signalling would have to be
per-thread.

As always, glad to provide entertainment to the list ;-)

--
Bojan

Attachments:

httpd-close_busy_read.patch (4.07 KB)

sf at sfritsch

Oct 18, 2009, 1:37 AM

Post #6 of 21 (2074 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sunday 18 October 2009, Bojan Smojver wrote:
> The idea here is that a busy server is highly unlikely to be stuck
> reading using all its children over a maintenance interval (i.e. we
> expect at least one of those readers to turn into something else
> during the interval).

An attacker can easily circumvent this by opening one connection that
slowly downloads a large file. Checking for a certain percentage of
all children would be better.

However, there is a real problem with all approaches that look for
SERVER_BUSY_READ: The attacker can just use a URL that accepts POST
requests and send the request body very slowly. These connections have
the state SERVER_BUSY_WRITE. This problem affects mod_antiloris and
mod_noloris, too (but not mod_reqtimeout).

Maybe another state SERVER_BUSY_READ_BODY could be introduced? Or the
state could be changed to SERVER_BUSY_READ again when the request body
is read? I haven't checked how difficult this would be, though.

bojan at rexursive

Oct 18, 2009, 1:48 AM

Post #7 of 21 (2075 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sun, 2009-10-18 at 10:37 +0200, Stefan Fritsch wrote:
> An attacker can easily circumvent this by opening one connection that
> slowly downloads a large file. Checking for a certain percentage of
> all children would be better.

Yeah, I was thinking of that too - just wanted to err on the side of
caution by having the lot in read state. We can also create a checksum
of pids/states and if this is the same second time around, pronounce it
under attack (combined with percentage).

> However, there is a real problem with all approaches that look for
> SERVER_BUSY_READ: The attacker can just use a URL that accepts POST
> requests and send the request body very slowly. These connections
> have
> the state SERVER_BUSY_WRITE. This problem affects mod_antiloris and
> mod_noloris, too (but not mod_reqtimeout).
>
> Maybe another state SERVER_BUSY_READ_BODY could be introduced? Or the
> state could be changed to SERVER_BUSY_READ again when the request
> body
> is read? I haven't checked how difficult this would be, though.

Yeah, that part sucks.

But the real deal is that there is a finite amount of resources we have
(be that sockets, threads or processes) and we need to decide how we are
willing to dole these out.

--
Bojan

sf at sfritsch

Oct 18, 2009, 2:15 AM

Post #8 of 21 (2076 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sunday 18 October 2009, Bojan Smojver wrote:
> But the real deal is that there is a finite amount of resources we
> have (be that sockets, threads or processes) and we need to decide
> how we are willing to dole these out.

What about defining an API to determine if the server has resource
shortage and then make various parts of Apache react to that, e.g.

- reduce KeepAliveTimeout
- increase MinRateLimit in mod_reqtimeout
- decreatse MaxClientConnections in mod_noloris
- increase minimum download rate requirement (mod_bwlimit?)
- kill connections in BUSY_READ state

Maybe this could be a two step process, e.g. start reduceing timeouts
when 80% of the threads are in use and start killing connections in
BUSY_READ or keepalive state when all threads are busy.

This would be a more general version of the patch posted here a few
months back, which dynamically adjusted Timeout depending on the
server load.

But I fear this would only be effective if there is a way to also
influence threads that are blocking on IO.

bojan at rexursive

Oct 18, 2009, 3:56 AM

Post #9 of 21 (2062 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sun, 2009-10-18 at 19:48 +1100, Bojan Smojver wrote:
> We can also create a checksum of pids/states and if this is the same
> second time around, pronounce it under attack

Like this.

--
Bojan

Attachments:

httpd-close_too_busy.patch (6.12 KB)

bojan at rexursive

Oct 18, 2009, 5:46 AM

Post #10 of 21 (2053 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sun, 2009-10-18 at 21:56 +1100, Bojan Smojver wrote:
> Like this.

Ah, details, details...

--
Bojan

Attachments:

httpd-close_too_busy.patch (5.96 KB)

bojan at rexursive

Oct 18, 2009, 5:53 AM

Post #11 of 21 (2059 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sun, 2009-10-18 at 23:46 +1100, Bojan Smojver wrote:
> + /* If after one maintenace interval we still see the same
> + * situation on the scoreboard, close 10% of client
> sockets.

The percentage picked out of thin air, of course.

--
Bojan

bojan at rexursive

Oct 18, 2009, 2:07 PM

Post #12 of 21 (2057 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sun, 2009-10-18 at 21:56 +1100, Bojan Smojver wrote:
> Like this.

Here is a more aggressive variant, which disconnects all sockets in read
state and at least 10% of all sockets when we get in trouble. On my test
machine, it is quite effective against slowloris.

General idea is that:

- a properly configured server will not be maxing out
- if maxing out does happen, scoreboard is bound to change within some
time period (which we can pick), or we are seeing an attack

With this approach (i.e. the scoreboard checksum), it doesn't matter
much whether we are being attacked by putting workers into
SERVER_BUSY_READ state or not.

--
Bojan

Attachments:

httpd-close_too_busy.patch (6.00 KB)

sf at sfritsch

Oct 18, 2009, 3:00 PM

Post #13 of 21 (2049 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Sunday 18 October 2009, Bojan Smojver wrote:
> - a properly configured server will not be maxing out
> - if maxing out does happen, scoreboard is bound to change within
> some time period (which we can pick), or we are seeing an attack
>
> With this approach (i.e. the scoreboard checksum), it doesn't
> matter much whether we are being attacked by putting workers into
> SERVER_BUSY_READ state or not.

Randomly killing possibly legitimate connections is some kind of DoS,
too. But it's probably better than the current behaviour. It could
even improve the situation in the case where many processes are stuck
waiting for a broken ldap/backend/etc. server.

In any case, you should try to kill workers with SERVER_BUSY_KEEPALIVE
before randomly killing processes.

bojan at rexursive

Oct 18, 2009, 3:22 PM

Post #14 of 21 (2047 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Mon, 2009-10-19 at 00:00 +0200, Stefan Fritsch wrote:
> Randomly killing possibly legitimate connections is some kind of DoS,
> too.

For sure. Anything we do, including reducing connection timeout on a
heavily loaded server is DoS. We just pick the type of DoS we want,
instead of the one attacker wants :-)

> But it's probably better than the current behaviour. It could
> even improve the situation in the case where many processes are stuck
> waiting for a broken ldap/backend/etc. server.

Yep, that's the deal.

> In any case, you should try to kill workers with SERVER_BUSY_KEEPALIVE
> before randomly killing processes.

True.

I meant the patch more as a proof of concept than a real solution (that
SIGINT is a poke in the eye and I haven't touched worker at all). I've
attached what you probably meant anyway.

--
Bojan

Attachments:

httpd-close_too_busy.patch (6.16 KB)

bojan at rexursive

Oct 18, 2009, 4:40 PM

Post #15 of 21 (2040 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Mon, 2009-10-19 at 09:22 +1100, Bojan Smojver wrote:
> that SIGINT is a poke in the eye

Maybe we could use SIGURG here. After all we do have an urgent message
for the socket :-)

--
Bojan

bojan at rexursive

Oct 18, 2009, 6:27 PM

Post #16 of 21 (2051 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Mon, 2009-10-19 at 10:40 +1100, Bojan Smojver wrote:
> Maybe we could use SIGURG here. After all we do have an urgent message
> for the socket :-)

Hmm, it's a real bummer we don't have more user defined signals. We
could reach for RT stuff, but I'm not sure how portable that is.

Anyhow, here is a version that uses the dreadful SIGINT a bit more
smartly. It'll only close client socket if SIGINT came from the parent
process (provided sigaction() is available). Otherwise, it'll exit the
process, as usual.

--
Bojan

Attachments:

httpd-close_too_busy.patch (7.08 KB)

bojan at rexursive

Oct 18, 2009, 7:01 PM

Post #17 of 21 (2032 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Mon, 2009-10-19 at 12:27 +1100, Bojan Smojver wrote:
> +#ifndef NO_USE_SIGACTION
> +static void close_client_socket(int sig, siginfo_t *info, void
> *context)
> +#else
> +static void close_client_socket(int sig)
> +#endif
> +{
> + if (client_socket != -1) {
> +#ifndef NO_USE_SIGACTION
> + if (info->si_pid == getppid()) {
> +#endif
> + close(client_socket);
> + client_socket = -1;
> +#ifndef NO_USE_SIGACTION
> + }
> + else {
> + clean_child_exit(0);
> + }
> +#endif
> + }
> +}

Actually, this should be:

+#ifndef NO_USE_SIGACTION
+static void close_client_socket(int sig, siginfo_t *info, void
*context)
+#else
+static void close_client_socket(int sig)
+#endif
+{
+#ifndef NO_USE_SIGACTION
+ if (info->si_pid == getppid()) {
+#endif
+ if (client_socket != -1) {
+ close(client_socket);
+ client_socket = -1;
+ }
+#ifndef NO_USE_SIGACTION
+ }
+ else {
+ clean_child_exit(0);
+ }
+#endif
+}

--
Bojan

bojan at rexursive

Oct 20, 2009, 3:43 PM

Post #18 of 21 (1965 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Mon, 2009-10-19 at 09:22 +1100, Bojan Smojver wrote:
> and I haven't touched worker at all

Here is another take on the whole thing, this time worker included. In
my tests, prefork performed a lot better with this approach. Worker
tended to remain under DoS more and would also disconnect far more
"good" connections than prefork. Probably something to do with me
closing wrong sockets - didn't have time to check in detail.

Patches combine two different approaches when we get maxed out:

1. If over 95% of workers are stuck in read, we close reader sockets.
2. If scoreboard hasn't changed during the maintenance interval, we
close all readers and at least 10% of all sockets.

--
Bojan

Attachments:

httpd-close_sockets.patch (17.3 KB)

bojan at rexursive

Oct 21, 2009, 1:17 AM

Post #19 of 21 (1954 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Wed, 2009-10-21 at 09:43 +1100, Bojan Smojver wrote:
> Probably something to do with me
> closing wrong sockets - didn't have time to check in detail.

Actually, calling close() is the wrong thing to do. Calling shutdown()
is the go.

--
Bojan

bojan at rexursive

Oct 21, 2009, 3:11 AM

Post #20 of 21 (1948 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Wed, 2009-10-21 at 19:17 +1100, Bojan Smojver wrote:
> Actually, calling close() is the wrong thing to do. Calling shutdown()
> is the go.

This is what I mean.

--
Bojan

Attachments:

httpd-shutdown_sockets.patch (17.4 KB)

bojan at rexursive

Nov 1, 2009, 5:49 PM

Post #21 of 21 (1733 views)
Permalink

Re: Crazy slowloris mitigation patch [In reply to]

On Fri, 2009-10-16 at 08:00 +1100, Bojan Smojver wrote:
> While playing with slowloris against prefork, I wrote the attached
> craziness.

On the back of the bug #48094, I reworked the latest version of this
craziness, which now suspends worker threads before shutting down the
sockets. Just in case someone wanted to play with it.

--
Bojan

Attachments:

httpd-shutdown_sockets.patch (17.9 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads