guy at trigger
Nov 13, 2002, 8:33 AM
Post #5 of 9
(1385 views)
Permalink
|
|
Corrupt request causes Denial of Service cascade
[In reply to]
|
|
Hello,
Thanks for the help Theo. I was not able to compile once the patch was
applied.
The r->files in the line below is not defined in my version
+ r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);
I have a CVS version from late last week. Did you perhaps mean r->filename?
I tried using r->filename for the undefined structure memeber, but the
cascade still starts immediately.
I have some more info though (when compiled with r->filename!):
1. The error message you added in the patch never gets triggered.
2. The looping seems to initially be a tight loop, looping after
"mod_backhand: Check 3 --" is displayed in the log. Eventually (when all the
children started?) the loop seems to grow to include the check 6 portion and
logs a failure after that point. I have included the transition point of the
logs below.
3. I tried adjusting the <LocationMatch> that is backhanding everything from
".*" to "/.+" to see if it would somehow affect the looping. With this
setting the cascade deos not start, but the child segfaults and exits. I
tried using gdb to extract some useful info from apache, but was unable to
acheive anything.
Dennis
=== Logfile Loop Transition ===
sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 3 --
sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 3 --
sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21
[Wed Nov 13 10:09:56 2002] [error] server reached MaxClients setting,
consider raising the MaxClients setting
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 6 -- headers(-1)
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Tried... failed
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 3 --
sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 6 -- headers(-1)
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Tried... failed
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 3 --
=== Segmentation Fault Example ===
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- UnixSocketDir set to
/var/backhand/backhand
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Broadcast
10.1.1.255:4445 added
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Multicast accept
10.1.1.0/24
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Multicast accept
10.1.1.0/24
[Wed Nov 13 10:32:51 2002] [notice] backhand_init(16615) spawning moderator
(PID 16616)
[Wed Nov 13 10:32:52 2002] [notice] waiting for mod_backhand moderator to
start
[Wed Nov 13 10:32:52 2002] [notice] mod_backhand moderator ready to go
[Wed Nov 13 10:32:52 2002] [notice] Apache/1.3.27 (Unix) mod_gzip/1.3.26.1a
mod_backhand/1.2.1 mod_ssl/2.8.12 OpenSSL/0.9.6g mod_jk/1.2.0 configured --
resuming normal operations
[Wed Nov 13 10:32:52 2002] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Wed Nov 13 10:33:09 2002] [notice] (Translate <=) = { /index.html, (null),
(null) }
[Wed Nov 13 10:33:09 2002] [notice] (Translate =>) = { /index.html, (null),
(null) }
[Wed Nov 13 10:33:09 2002] [notice] (Type <=) = { (null), /index.html,
(null) }
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:33:09 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/index.html, backhand/redirect }
[Wed Nov 13 10:33:09 2002] [notice] (Translate <=) = {
backhand:/backhand:/index.html, (null), (null) }
[Wed Nov 13 10:33:09 2002] [notice] child pid 16618 exit signal Segmentation
fault (11)
-----Original Message-----
From: backhand-users-admin [at] lists
[mailto:backhand-users-admin [at] lists]On Behalf Of Theo
Schlossnagle
Sent: Wednesday, November 13, 2002 12:39 AM
To: backhand-users [at] lists
Cc: Theo Schlossnagle
Subject: Re: [m_b_users] Corrupt request causes Denial of Service
cascade
On Sunday, Nov 10, 2002, at 14:28 US/Eastern, Dennis Cartier wrote:
Okay. I have a very similar set up to you on this end sans the
mod_gzip. I have mod_backhand/mod_ssl/mod_log_spread/mod_rewrite all
working smoothly together (mod_rewrite + mod_backhand is a pain in the
ass). Anyway, I attempted to replicate you error by issuing a request
like: "/lalala.html" to my server (no GET or HTTP/1.0) to see if I
could "throw it through a loop". Alas, I got the expected error back
-- being "malformed request" and everything was fine.
However, the next morning, I was pages about Spread problems on that
machine and logged into fine the load on its way up from 2 to 158. I
did notice two things
(1) one of the back-end machines was causing a front-end mod_backhand
error. As the load was 158 when I logged in there was about zero
troubleshooting I could do. But I did have quite a few: "could not get
valid connection -- forced local" in my error log, which would lead me
to believe that there is some type of loop.
(2) Spread was hosed. I could spuser in, but when I joined a group it
took over 20 seconds to return my membership message -- even on the
other machines in the cluster with loads between 0 and 1.
So, my guess is that it may be caused by (1), but the several severe
problems I have had in the (2) arena have me a little confused as to
which is the cause and which the effect. For the case of (1), I have
patched mod_backhand.c to try to prevent those loops (and detect them).
If the patch works, I will commit it to CVS. For the bold, a patch of
these changes (against mod_backhand.c revision 1.50) is below.
--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------
RCS file: /storage/cvs/jesus/mod_backhand/mod_backhand.c,v
retrieving revision 1.50
diff -u -3 -r1.50 mod_backhand.c
--- mod_backhand.c 13 Sep 2002 00:40:42 -0000 1.50
+++ mod_backhand.c 13 Nov 2002 05:35:16 -0000
@@ -913,6 +913,11 @@
ap_table_setn((r->main)?r->main->notes:r->notes,
"BackhandHandlerExecuted", "Done");
ap_internal_redirect(r->uri, r);
+ } else {
+ ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
+ "mod_backhand: "
+ "attempted mod_backhand redirection twice. BUG.");
+ return DECLINED;
}
if(loglevel & MBLL_MBCS3) {
ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
@@ -1335,7 +1340,7 @@
const char *rhost = ap_table_get(subr->notes,
"Backhand-Redirect-Host");
if(rhost) ap_table_set(r->notes, "Backhand-Redirect-Host",
rhost);
r->uri = ap_pstrcat(r->pool, "backhand:", r->uri, NULL);
- r->filename = ap_pstrdup(r->pool, r->uri);
+ r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);
r->handler = "backhand-redirection";
if(loglevel & MBLL_DCSN1) {
ap_log_error(APLOG_MARK, APLOG_NOTICE|APLOG_NOERRNO, NULL,
@@ -1379,6 +1384,8 @@
return DECLINED;
}
memmove(r->uri, &r->uri[9], strlen(&r->uri[9])+1);
+ if(strncmp(r->filename, "backhand:", 9)==0)
+ memmove(r->filename, &r->filename[9], strlen(&r->filename[9])+1);
return OK;
}
_______________________________________________
backhand-users mailing list
backhand-users [at] lists
http://lists.backhand.org/mailman/listinfo/backhand-users
|
