Mailing List Archive: Apache: Dev

proxy failover/load balance

Index | Next | Previous | View Threaded

robby.pedrica at brand-id

Jan 30, 2006, 12:26 AM

Post #1 of 19 (16293 views)
Permalink

proxy failover/load balance

wrt information provided by Mladen Turk, Jim Jagielski and Andreas Wieczorek:

I'm having issues when using mod_proxy/mod_proxy_balancer and it appears these have been alluded to on the dev mailing list:

1. apache 220 doesn't appear to use the status=d value for a balancer member when loading ( status=disabled doesn't work as apache complains about the syntax ); when starting with status=d, apache starts fine but balancer manager still indicates that host you set as disabled in config is actually available
2. when manually disabling a member in the balancer manager, it comes back online automatically after about a minute or 2
3. I've tried sticky session id ( CSPSESSIONID / CSPCHD ), redirect with status=d and redirect with route but in all cases, url requests are load balanced between the 2 backend hosts. I actually want sessions to either stay on one host ( dynamic data to a backend intersystems cache server ) or have one host disabled while the other serves ...

Any help would be appreciated.

Robby Pedrica

ruediger.pluem at vodafone

Jan 31, 2006, 2:43 AM

Post #2 of 19 (15984 views)
Permalink

AW: proxy failover/load balance [In reply to]

-----Ursprüngliche Nachricht-----
Von: Robby Pedrica
Gesendet: Montag, 30. Januar 2006 09:27
Betreff: proxy failover/load balance

> wrt information provided by Mladen Turk, Jim Jagielski and Andreas Wieczorek:
>
> I'm having issues when using mod_proxy/mod_proxy_balancer and it appears these have been alluded to on the dev mailing list:
>
> 1. apache 220 doesn't appear to use the status=d value for a balancer member when loading ( status=disabled doesn't work as apache complains about the

This seems to be a bug in set_worker_param of mod_proxy.c. It should exit the for loop once it found a valid thing.

> syntax ); when starting with status=d, apache starts fine but balancer manager still indicates that host you set as disabled in config is actually
> available

This seems to be a bug in init_balancer_members of mod_proxy_balancer.c which seems to overwrite the status of the balancer members with
PROXY_WORKER_INITIALIZED.

> 2. when manually disabling a member in the balancer manager, it comes back online automatically after about a minute or 2

What do you mean with online? Is it enabled again? If yes, this might be related to the bug above as init_balancer_members is called
by child_init which runs in the child_init hook. So I think everytime httpd creates a new child process this gets reseted.

Regards

Rüdiger

robby.pedrica at brand-id

Jan 31, 2006, 11:13 AM

Post #3 of 19 (15987 views)
Permalink

Re: AW: proxy failover/load balance [In reply to]

What do you mean with online? Is it enabled again? If yes, this might be related to the bug above as init_balancer_members is called by child_init which runs in the child_init hook. So I think everytime httpd creates a new child process this gets reseted. Regards Rüdiger

Hi Rudiger

Thanks for the reply:

a. any idea on when a fix will be available or whether there are patches available for these items?
b. Yes, if you manually disable a member in balancer-manager, then it becomes enabled ( OK ) after a minute or 2

Regards

Robby

rpluem at apache

Jan 31, 2006, 1:32 PM

Post #4 of 19 (15972 views)
Permalink

Re: AW: proxy failover/load balance [In reply to]

On 01/31/2006 08:13 PM, Robby Pedrica wrote:
> Hi Rudiger
>
> Thanks for the reply:
>
> a. any idea on when a fix will be available or whether there are patches
> available for these items?
> b. Yes, if you manually disable a member in balancer-manager, then it becomes
> enabled ( OK ) after a minute or 2
>
> Regards
>
> Robby

Could you please give the attached patch a try and let me know the results.

Regards

Rüdiger

Attachments:

worker_disabled.diff (2.00 KB)

robby.pedrica at brand-id

Feb 1, 2006, 1:32 AM

Post #5 of 19 (15982 views)
Permalink

Re: AW: proxy failover/load balance [In reply to]

Ruediger Pluem wrote:

On 01/31/2006 08:13 PM, Robby Pedrica wrote:
Hi Rudiger Thanks for the reply: a. any idea on when a fix will be available or whether there are patches available for these items? b. Yes, if you manually disable a member in balancer-manager, then it becomes enabled ( OK ) after a minute or 2 Regards Robby
Could you please give the attached patch a try and let me know the results. Regards Rüdiger

Hi Rudiger,

I've applied patches and recompiled. My results are as follows:

1. apache starts up with the member 'b' disabled now
2. if I shutdown the working member 'a' httpd, then manager shows the change only when you try and access the web site. Moves from state ok to err
3. If I bring the member 'a' back up again ( start httpd ) then manager shows the move from err to ok only after trying to access the site.

This is definitely an improvement even if failover isn't working because at least we can manually failover ...

Thanks Rudiger!!!

Regards, Robby

ruediger.pluem at vodafone

Feb 1, 2006, 1:52 AM

Post #6 of 19 (15955 views)
Permalink

AW: AW: proxy failover/load balance [In reply to]

-----Ursprüngliche Nachricht-----
Von: Robby Pedrica

> Hi Rudiger,
>
> I've applied patches and recompiled. My results are as follows:
>
> 1. apache starts up with the member 'b' disabled now
> 2. if I shutdown the working member 'a' httpd, then manager shows the change only when you try and access the web site. Moves from state ok to err
> 3. If I bring the member 'a' back up again ( start httpd ) then manager shows the move from err to ok only after trying to access the site.

2. and 3. work as designed. httpd only changes the status for a worker to err if a request failed in the middle or if it tried to assign a request
to this worker and that failed. Once the worker is in error state httpd will not try to assign a new request to this worker for "retry" seconds
(see parameter explanation at http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass). After that grace period the worker gets new requests
assigned again and will switch back to ok once it processed the assigned request successfully.

>
> This is definitely an improvement even if failover isn't working because at least we can manually failover ...

Considering my explanation above, what do you mean by "failover isn't working"?

Regards

Rüdiger

robby.pedrica at brand-id

Feb 1, 2006, 1:55 AM

Post #7 of 19 (15997 views)
Permalink

Re: AW: AW: proxy failover/load balance [In reply to]

Plüm wrote:

-----Ursprüngliche Nachricht----- Von: Robby Pedrica
Hi Rudiger, I've applied patches and recompiled. My results are as follows: 1. apache starts up with the member 'b' disabled now 2. if I shutdown the working member 'a' httpd, then manager shows the change only when you try and access the web site. Moves from state ok to err 3. If I bring the member 'a' back up again ( start httpd ) then manager shows the move from err to ok only after trying to access the site.
2. and 3. work as designed. httpd only changes the status for a worker to err if a request failed in the middle or if it tried to assign a request to this worker and that failed. Once the worker is in error state httpd will not try to assign a new request to this worker for "retry" seconds (see parameter explanation at http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass"]http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass). After that grace period the worker gets new requests assigned again and will switch back to ok once it processed the assigned request successfully.
This is definitely an improvement even if failover isn't working because at least we can manually failover ...
Considering my explanation above, what do you mean by "failover isn't working"? Regards Rüdiger

Thanks Rudiger,

I've set redirect and route values - not sure if these are correct:

<Proxy balancer://mycluster>
BalancerMember http://192.168.4.2:80"]http://192.168.4.2:80 redirect=1
BalancerMember http://192.168.4.3:80"]http://192.168.4.3:80 route=1 status=D
</Proxy>

I'm assuming that if member a. fails then requests will be redirected to member b. - this is not happening currently.

Regards

Robby

ruediger.pluem at vodafone

Feb 1, 2006, 2:24 AM

Post #8 of 19 (15986 views)
Permalink

AW: AW: AW: proxy failover/load balance [In reply to]

-----Ursprüngliche Nachricht-----
Von: Robby Pedrica

> Thanks Rudiger,
>
> I've set redirect and route values - not sure if these are correct:
>
> <Proxy balancer://mycluster>
> BalancerMember http://192.168.4.2:80 redirect=1
> BalancerMember http://192.168.4.3:80 route=1 status=D
> </Proxy>
>
> I'm assuming that if member a. fails then requests will be redirected to member b. - this is not happening currently.

This only works with session stickyness. So

ProxyPass / balancer://mycluster stickysession=SESSION_COOKIE
<Proxy balancer://mycluster>
BalancerMember http://192.168.4.2:80 route=a redirect=b
BalancerMember http://192.168.4.3:80 route=b status=Disabled
</Proxy>

Furthermore your backend should deliver a session cookie

SESSION_COOKIE which has .a / .b added to the session id.
If you don't have an application on the backend that does this (from previous statements
I assume that you are using a httpd as backend) you can use
mod_headers on the backend servers to set these cookies.

Header add Set-Cookie "SESSION_COOKIE=anything.a;Path=/"

Regards

Rüdiger

ruediger.pluem at vodafone

Feb 1, 2006, 3:03 AM

Post #9 of 19 (15992 views)
Permalink

AW: AW: AW: proxy failover/load balance [In reply to]

> -----Ursprüngliche Nachricht-----
> Von: Plüm, Rüdiger,
>
> This only works with session stickyness. So
>
> ProxyPass / balancer://mycluster stickysession=SESSION_COOKIE
> <Proxy balancer://mycluster> BalancerMember
> http://192.168.4.2:80 route=a redirect=b BalancerMember
> http://192.168.4.3:80 route=b status=Disabled </Proxy>
>
> Furthermore your backend should deliver a session cookie
>
> SESSION_COOKIE which has .a / .b added to the session id.
> If you don't have an application on the backend that does
> this (from previous statements I assume that you are using a
> httpd as backend) you can use mod_headers on the backend
> servers to set these cookies.
>
> Header add Set-Cookie "SESSION_COOKIE=anything.a;Path=/"

I fear that the configuration above is not sufficient without the attached
patch.

Regards

Rüdiger

Attachments:

disabled_worker.diff (1.19 KB)

jim at jaguNET

Feb 1, 2006, 5:42 AM

Post #10 of 19 (15975 views)
Permalink

Re: AW: AW: proxy failover/load balance [In reply to]

I think we're in agreement that the current "failover" does
not work as it should with HTTP, and is quite
cumbersome to get it to work. :)

I hope to later on this week work on code that has
a real "hot standby" status, and avoids the requirement
for sticky sessions. It won't replace what's in
there now (for AJP) but will make it easier
to implement failover for simple tasks.

jim at jaguNET

Feb 1, 2006, 5:57 AM

Post #11 of 19 (15976 views)
Permalink

Re: AW: proxy failover/load balance [In reply to]

Why the breaks? Certainly we still want to continue the
for loop even if we see a valid setting. For example,
to set a worker in DISABLED and STOPPED mode.

On Jan 31, 2006, at 4:32 PM, Ruediger Pluem wrote:
>
> Index: modules/proxy/mod_proxy.c
> ===================================================================
> --- modules/proxy/mod_proxy.c (Revision 371134)
> +++ modules/proxy/mod_proxy.c (Arbeitskopie)
> @@ -200,18 +200,21 @@
> worker->status |= PROXY_WORKER_DISABLED;
> else
> worker->status &= ~PROXY_WORKER_DISABLED;
> + break;
> }
> else if (*v == 'S' || *v == 's') {
> if (mode)
> worker->status |= PROXY_WORKER_STOPPED;
> else
> worker->status &= ~PROXY_WORKER_STOPPED;
> + break;
> }
> else if (*v == 'E' || *v == 'e') {
> if (mode)
> worker->status |= PROXY_WORKER_IN_ERROR;
> else
> worker->status &= ~PROXY_WORKER_IN_ERROR;
> + break;
> }
> else {
> return "Unknow status parameter option";

jim at jaguNET

Feb 1, 2006, 6:01 AM

Post #12 of 19 (15990 views)
Permalink

Re: AW: proxy failover/load balance [In reply to]

I mean, of course, having status be simple flags +d+s
for example, rather than the whole word. The code looks
to be designed with that in mind.

On Feb 1, 2006, at 8:57 AM, Jim Jagielski wrote:

>
> Why the breaks? Certainly we still want to continue the
> for loop even if we see a valid setting. For example,
> to set a worker in DISABLED and STOPPED mode.
>
> On Jan 31, 2006, at 4:32 PM, Ruediger Pluem wrote:
>>
>> Index: modules/proxy/mod_proxy.c
>> ===================================================================
>> --- modules/proxy/mod_proxy.c (Revision 371134)
>> +++ modules/proxy/mod_proxy.c (Arbeitskopie)
>> @@ -200,18 +200,21 @@
>> worker->status |= PROXY_WORKER_DISABLED;
>> else
>> worker->status &= ~PROXY_WORKER_DISABLED;
>> + break;
>> }
>> else if (*v == 'S' || *v == 's') {
>> if (mode)
>> worker->status |= PROXY_WORKER_STOPPED;
>> else
>> worker->status &= ~PROXY_WORKER_STOPPED;
>> + break;
>> }
>> else if (*v == 'E' || *v == 'e') {
>> if (mode)
>> worker->status |= PROXY_WORKER_IN_ERROR;
>> else
>> worker->status &= ~PROXY_WORKER_IN_ERROR;
>> + break;
>> }
>> else {
>> return "Unknow status parameter option";
>

ruediger.pluem at vodafone

Feb 1, 2006, 6:02 AM

Post #13 of 19 (15974 views)
Permalink

AW: AW: AW: proxy failover/load balance [In reply to]

> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
>
> I think we're in agreement that the current "failover" does
> not work as it should with HTTP, and is quite
> cumbersome to get it to work. :)

Apart from the fact that it currently does not even work without
patches :-).
So I am keen on feedback by Robby. I hope to find time to commit these
changes to the trunk tonight, so that it works at least in the cumbersome way :-).

>
> I hope to later on this week work on code that has
> a real "hot standby" status, and avoids the requirement
> for sticky sessions. It won't replace what's in
> there now (for AJP) but will make it easier
> to implement failover for simple tasks.
>

Sounds good.

Regards

Rüdiger

ruediger.pluem at vodafone

Feb 1, 2006, 6:15 AM

Post #14 of 19 (15974 views)
Permalink

AW: AW: proxy failover/load balance [In reply to]

> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
>
> Why the breaks? Certainly we still want to continue the
> for loop even if we see a valid setting. For example,
> to set a worker in DISABLED and STOPPED mode.

1. Currently there is no clear separation letter.
2. Setting status=disabled will result in "Unknow status parameter option"

But maybe I just understand the syntax of status wrong and status should
not be compiled of clear text words but rather of something cryptic like

[eEdDsS\+-]+

where

e,E is error
d,D is disabled
s,S is stopped

But as it is not documented I don't know :-)

Regards

Rüdiger

jim at jaguNET

Feb 1, 2006, 6:18 AM

Post #15 of 19 (15966 views)
Permalink

Re: AW: AW: AW: proxy failover/load balance [In reply to]

On Feb 1, 2006, at 9:02 AM, Plüm, Rüdiger, VIS wrote:

>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jim Jagielski
>>
>> I think we're in agreement that the current "failover" does
>> not work as it should with HTTP, and is quite
>> cumbersome to get it to work. :)
>
> Apart from the fact that it currently does not even work without
> patches :-).

Agreed :) Hence the quotes around failover ;)

> So I am keen on feedback by Robby. I hope to find time to commit these
> changes to the trunk tonight, so that it works at least in the
> cumbersome way :-).
>

++1 !

ruediger.pluem at vodafone

Feb 1, 2006, 6:34 AM

Post #16 of 19 (15968 views)
Permalink

AW: AW: AW: AW: proxy failover/load balance [In reply to]

> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
>
> On Feb 1, 2006, at 9:02 AM, Plüm, Rüdiger, VIS wrote:

> > So I am keen on feedback by Robby. I hope to find time to
> commit these
> > changes to the trunk tonight, so that it works at least in the
> > cumbersome way :-).
> >

I will cut the breaks from the patch to keep the current syntax alive.
The correct syntax should be defined and discussed later.
And of course documented, once there is a decision on that :-).

Regards

Rüdiger

robby.pedrica at brand-id

Feb 1, 2006, 8:54 AM

Post #17 of 19 (15978 views)
Permalink

Re: AW: AW: AW: AW: proxy failover/load balance [In reply to]

Plüm wrote:

-----Ursprüngliche Nachricht----- Von: Jim Jagielski On Feb 1, 2006, at 9:02 AM, Plüm, Rüdiger, VIS wrote:

So I am keen on feedback by Robby. I hope to find time to
commit these
changes to the trunk tonight, so that it works at least in the cumbersome way :-).

I will cut the breaks from the patch to keep the current syntax alive. The correct syntax should be defined and discussed later. And of course documented, once there is a decision on that :-). Regards Rüdiger

Hi Rudiger,

These list entries have been coming think and fast ; ) A little more info on our setup:

We have two apache 2.2 servers in a heartbeat hot-standby pair, acting as the proxies. Therefore if one of the proxies goes down, HA switches requests to the other. So we're actually using a virtual ip between the 2 proxies. The 2 back end servers run apache 1.3 with Intersystems CACHE and their CSP app server implementation. Requests are basically sent through a cgi sitting in apache to Cache ...

I'm not sure how to incorporate mod_header as you've indicated but will have a chat to the programmers tomorrow and see what we can do. Our requirement from mod_proxy/mod_proxy_balancer is basically to have one proxy receiving all requests and if it goes down, then switch to the next proxy, etc.

As per my previous post, I can start apache now with the one worker disabled. However, the failover is not occurring. I understand this is very bleeding edge so ... thanks for all the help so far.

Regards

Robby

rpluem at apache

Feb 4, 2006, 2:08 PM

Post #18 of 19 (15982 views)
Permalink

Re: proxy failover/load balance [In reply to]

On 02/01/2006 03:34 PM, Plüm wrote:

>>
>>>changes to the trunk tonight, so that it works at least in the
>>>cumbersome way :-).
>>>
>
>
> I will cut the breaks from the patch to keep the current syntax alive.
> The correct syntax should be defined and discussed later.
> And of course documented, once there is a decision on that :-).

Finally I found some time to continue my work on this problem.
As of now I only committed the change that prevents mod_proxy_balancer
from overwriting the status of workers when creating a new child process
(http://svn.apache.org/viewcvs?rev=374929&view=rev).
As announced I left the IMHO somewhat cryptic syntax to predefine the status
of a worker in the configuration as it is. So to disable a worker at start
status=disabled cannot be used, but status=d must be used instead.

Even more important I did no further changes which means that automatic
failover as wanted by Robby still does not work.
During my search for a solution of this I got more and more unsure of what
is the desired behaviour of what configuration. So I would like to start
a discussion here.

I see at least the following questions:

1. What is the exact meaning of a disabled worker? Should it be considered
for use if find_best_worker cannot find a worker in non error mode and nofailover
is set to off?
Should a redirected worker that is disabled be used?

2. What about chains of redirects (worker 'a' has a redirect to worker 'b',
worker 'b' has a redirect to worker 'c')?

Regards

Rüdiger

robby.pedrica at brand-id

Feb 4, 2006, 11:34 PM

Post #19 of 19 (15992 views)
Permalink

Re: proxy failover/load balance [In reply to]

Ruediger Pluem wrote:
> Finally I found some time to continue my work on this problem.
> As of now I only committed the change that prevents mod_proxy_balancer
> from overwriting the status of workers when creating a new child process
> (http://svn.apache.org/viewcvs?rev=374929&view=rev).
> As announced I left the IMHO somewhat cryptic syntax to predefine the status
> of a worker in the configuration as it is. So to disable a worker at start
> status=disabled cannot be used, but status=d must be used instead.
>
> Even more important I did no further changes which means that automatic
> failover as wanted by Robby still does not work.
> During my search for a solution of this I got more and more unsure of what
> is the desired behaviour of what configuration. So I would like to start
> a discussion here.
>
> I see at least the following questions:
>
> 1. What is the exact meaning of a disabled worker? Should it be considered
> for use if find_best_worker cannot find a worker in non error mode and nofailover
> is set to off?
> Should a redirected worker that is disabled be used?
>
> 2. What about chains of redirects (worker 'a' has a redirect to worker 'b',
> worker 'b' has a redirect to worker 'c')?
>
>
> Regards
>
> Rüdiger
>
>
>
Hi Rudiger,

Thanks for the info. My thoughts on the options would be as follows:

(ok) enabled - a functional and processing worker
(dis) disabled - a worked disabled manually or by some condition eg. 5
workers but you only require 4 to process requests ( not sure how useful
this is )
(err) error - a worker disabled due to a fault in the backend, recheck
after timeout period
(stdby) standby - an enabled worker not currently receiving any requests
eg. all requests go node a. - if node a. fails over then use node b. -
this would sort of solve the current inability for sessions to stay on
one node ( if you don't have specific app session support )

Standby mode would be enough if you don't have session support
specifically, as the odds might be for eg. 1000 requests a day with an
up time of 100 days = 1 in 100,000 requests that might have an issue; I
can accept those odds ; ) Not sure which is easier, building generic
session support or a hot standby mode.

1. as above, not sure how useful this is and yes, if the job of a
disabled worker is to assume work when required, then I'd use it. Maybe
standby might be better definition.
2. not sure how difficult the logic would be, but I'd just go for
redirect a and/or b to c. Rudiger, what's your reasoning for chaining as
per no. 2 above?

Apologies I can't help more programatically ... my programming is baaaaad.

Regards. Robby.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com