Mailing List Archive: Linux Virtual Server: Users

[lvs-users] ldirectord feature patch - add abilility to signal system maintenance

Index | Next | Previous | View Threaded

sean at bruenor

May 14, 2009, 2:35 PM

Post #1 of 13 (2120 views)
Permalink

[lvs-users] ldirectord feature patch - add abilility to signal system maintenance

Patch 4/4

This patch adds the ability to notify ldirectord that a system should be
forced down for maintenance (or other reasons) even though the service
check passes.

This patch introduces a new option "maintenancedir" which allows the
administrator to specify a directory in which ldirectord will check
for the existence of files indicating if the realserver should be
forcibly set down (irrespective of and bypassing the normal service
check).

If a site is using a management tool such as cfengine or Puppet that
watches and manages the ldirectord configuration file it may not be
possible or practical to modify the configuration file to remove a
realserver or weight it to 0 during maintenance. This mechanism
provides a quick and simple way to signal ldirectord that a real server
should be set down without having to modify any configuration files.

Thanks,
Sean

Attachments:

ldirectord-maintenancedir.patch (5.66 KB)

horms at verge

May 14, 2009, 4:37 PM

Post #2 of 13 (2053 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Thu, May 14, 2009 at 05:35:40PM -0400, Sean Millichamp wrote:
> Patch 4/4
>
> This patch adds the ability to notify ldirectord that a system should be
> forced down for maintenance (or other reasons) even though the service
> check passes.
>
> This patch introduces a new option "maintenancedir" which allows the
> administrator to specify a directory in which ldirectord will check
> for the existence of files indicating if the realserver should be
> forcibly set down (irrespective of and bypassing the normal service
> check).
>
> If a site is using a management tool such as cfengine or Puppet that
> watches and manages the ldirectord configuration file it may not be
> possible or practical to modify the configuration file to remove a
> realserver or weight it to 0 during maintenance. This mechanism
> provides a quick and simple way to signal ldirectord that a real server
> should be set down without having to modify any configuration files.

This looks good to me. Though I wonder if it might be
better to save the results of ld_gethostbyaddr(), perhaps
at initialisation time. Then again, perhaps not as that
would stop ldirectord from respecting any DNS changes that
may occur.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

eric.robinson at psmnv

May 15, 2009, 5:40 AM

Post #3 of 13 (2034 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

When an RS is set down in this manner, does ldirectord tell LVS to stop
redirecting new connections but keep forwarding packets on existing
connections? Or does it just immediately cut off all communication with
the RS, which would have the effect of freezing connections for users
that are already on the RS?

--
Eric Robinson
Director of Information Technology
Physician Select Management, LLC
775.885.2211 x 111

-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver
[mailto:lvs-users-bounces [at] linuxvirtualserver] On Behalf Of Sean
Millichamp
Sent: Thursday, May 14, 2009 2:36 PM
To: lvs-users [at] linuxvirtualserver
Cc: Simon Horman
Subject: [lvs-users] ldirectord feature patch - add abilility to signal
system maintenance

Patch 4/4

This patch adds the ability to notify ldirectord that a system should be
forced down for maintenance (or other reasons) even though the service
check passes.

This patch introduces a new option "maintenancedir" which allows the
administrator to specify a directory in which ldirectord will check for
the existence of files indicating if the realserver should be forcibly
set down (irrespective of and bypassing the normal service check).

If a site is using a management tool such as cfengine or Puppet that
watches and manages the ldirectord configuration file it may not be
possible or practical to modify the configuration file to remove a
realserver or weight it to 0 during maintenance. This mechanism
provides a quick and simple way to signal ldirectord that a real server
should be set down without having to modify any configuration files.

Thanks,
Sean

Disclaimer - May 15, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.,Simon Horman. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

geoff at thehive

May 15, 2009, 6:19 AM

Post #4 of 13 (2045 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

we actually use the existing infrastructure for this. service checks
go to a script that returns OK if the system is available for new
connections, or doesn't if it is down

for our mysql checks we do something like 'select * from
ldirectord_pool where enabled=1' and truncate those tables when we're
taking a box out for maint.

however - it would be nice if we could do something like continue
routing old connections... but that would require something in terms
of ipvs level support I'm thinking

Geoff Harrison
Chief Architect
The Hive
(617) 301-6200

On May 14, 2009, at 5:35 PM, Sean Millichamp wrote:

> Patch 4/4
>
> This patch adds the ability to notify ldirectord that a system
> should be
> forced down for maintenance (or other reasons) even though the service
> check passes.
>
> This patch introduces a new option "maintenancedir" which allows the
> administrator to specify a directory in which ldirectord will check
> for the existence of files indicating if the realserver should be
> forcibly set down (irrespective of and bypassing the normal service
> check).
>
> If a site is using a management tool such as cfengine or Puppet that
> watches and manages the ldirectord configuration file it may not be
> possible or practical to modify the configuration file to remove a
> realserver or weight it to 0 during maintenance. This mechanism
> provides a quick and simple way to signal ldirectord that a real
> server
> should be set down without having to modify any configuration files.
>
> Thanks,
> Sean
> <ldirectord-
> maintenancedir.patch>_______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sean at bruenor

May 15, 2009, 8:45 AM

Post #5 of 13 (2042 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Fri, 2009-05-15 at 09:37 +1000, Simon Horman wrote:

> On Thu, May 14, 2009 at 05:35:40PM -0400, Sean Millichamp wrote:
> > This patch adds the ability to notify ldirectord that a system should be
> > forced down for maintenance (or other reasons) even though the service
> > check passes.

> This looks good to me. Though I wonder if it might be
> better to save the results of ld_gethostbyaddr(), perhaps
> at initialisation time. Then again, perhaps not as that
> would stop ldirectord from respecting any DNS changes that
> may occur.

I briefly pondered that. DNS already has opportunities for caching
external to ldirectord: you can run a local caching resolver with
appropriate TTLs or you can run nscd. It seems like a lot of added
complexity for something which could be solved already in another way.

I suspect any site requiring the level of control this option provides
will have the ability and experience to know how to handle the DNS
lookup problem via a local caching resolver or nscd. If this option
isn't enabled it becomes a non-issue as the DNS code never gets called.

Sean

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sean at bruenor

May 15, 2009, 9:04 AM

Post #6 of 13 (2033 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Fri, 2009-05-15 at 09:19 -0400, Geoff Harrison wrote:
> we actually use the existing infrastructure for this. service checks
> go to a script that returns OK if the system is available for new
> connections, or doesn't if it is down
>
> for our mysql checks we do something like 'select * from
> ldirectord_pool where enabled=1' and truncate those tables when we're
> taking a box out for maint.

True. However, while there are things I like in that approach I was
hoping to be able to continue using the built-in checks in ldirectord.
They are good enough for what we are currently doing and it (overall)
keeps things simpler.

My understanding it that you can't (for instance) run both a connect or
negotiate check and an external check for a single virtual service.

> however - it would be nice if we could do something like continue
> routing old connections... but that would require something in terms
> of ipvs level support I'm thinking

Isn't that what quiescent=yes is for? At least in masquerade mode (what
we use), when the weight is set to 0 ipvs continues to send existing TCP
sessions to that server. We weight a server to 0, then watch to make
sure all of the connections have closed, then bring the service down.

Sean

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

May 15, 2009, 9:24 AM

Post #7 of 13 (2035 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Fri, May 15, 2009 at 12:04:43PM -0400, Sean Millichamp wrote:
> On Fri, 2009-05-15 at 09:19 -0400, Geoff Harrison wrote:
> > we actually use the existing infrastructure for this. service checks
> > go to a script that returns OK if the system is available for new
> > connections, or doesn't if it is down
> >
> > for our mysql checks we do something like 'select * from
> > ldirectord_pool where enabled=1' and truncate those tables when we're
> > taking a box out for maint.
>
> True. However, while there are things I like in that approach I was
> hoping to be able to continue using the built-in checks in ldirectord.
> They are good enough for what we are currently doing and it (overall)
> keeps things simpler.
>
> My understanding it that you can't (for instance) run both a connect or
> negotiate check and an external check for a single virtual service.
>
> > however - it would be nice if we could do something like continue
> > routing old connections... but that would require something in terms
> > of ipvs level support I'm thinking
>
> Isn't that what quiescent=yes is for? At least in masquerade mode (what
> we use), when the weight is set to 0 ipvs continues to send existing TCP
> sessions to that server. We weight a server to 0, then watch to make
> sure all of the connections have closed, then bring the service down.

Just to clarify, quiescent=yes should work this way for all of LVS's
forwarding methods.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

chibi at gol

May 19, 2009, 12:27 AM

Post #8 of 13 (1988 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

Hello,

On Sat, 16 May 2009 02:24:55 +1000 Simon Horman wrote:
> > On Fri, 2009-05-15 at 09:19 -0400, Geoff Harrison wrote:
> > Isn't that what quiescent=yes is for? At least in masquerade mode
> > (what we use), when the weight is set to 0 ipvs continues to send
> > existing TCP sessions to that server. We weight a server to 0, then
> > watch to make sure all of the connections have closed, then bring the
> > service down.
>
> Just to clarify, quiescent=yes should work this way for all of LVS's
> forwarding methods.
>
It does not for SH (and DH I presume but I never tested that). Well, not as
expected at least. As I've been pointing out for well over a year and just
re-tested with the latest kernel.

If the weight goes to 0 with SH and quiescent=yes set connections will
keep going to the old server as expected. Alas even when they expire they
will keep being sent there as will completely new connections if the hash
result assigns them to the weight 0 RS.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi [at] gol Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

May 19, 2009, 2:52 AM

Post #9 of 13 (1989 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Tue, May 19, 2009 at 04:27:19PM +0900, Christian Balzer wrote:
>
> Hello,
>
> On Sat, 16 May 2009 02:24:55 +1000 Simon Horman wrote:
> > > On Fri, 2009-05-15 at 09:19 -0400, Geoff Harrison wrote:
> > > Isn't that what quiescent=yes is for? At least in masquerade mode
> > > (what we use), when the weight is set to 0 ipvs continues to send
> > > existing TCP sessions to that server. We weight a server to 0, then
> > > watch to make sure all of the connections have closed, then bring the
> > > service down.
> >
> > Just to clarify, quiescent=yes should work this way for all of LVS's
> > forwarding methods.
> >
> It does not for SH (and DH I presume but I never tested that). Well, not as
> expected at least. As I've been pointing out for well over a year and just
> re-tested with the latest kernel.
>
> If the weight goes to 0 with SH and quiescent=yes set connections will
> keep going to the old server as expected. Alas even when they expire they
> will keep being sent there as will completely new connections if the hash
> result assigns them to the weight 0 RS.

I don't seem to be able to reproduce that problem.
Do you have persistence set? If so, could you try setting
/proc/sys/net/ipv4/vs/expire_quiescent_template to 1 ?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

chibi at gol

May 19, 2009, 8:30 PM

Post #10 of 13 (1981 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

Hello Simon,

On Tue, 19 May 2009 19:52:42 +1000 Simon Horman wrote:

> On Tue, May 19, 2009 at 04:27:19PM +0900, Christian Balzer wrote:
> >
> > Hello,
> >
> > On Sat, 16 May 2009 02:24:55 +1000 Simon Horman wrote:
> > > > On Fri, 2009-05-15 at 09:19 -0400, Geoff Harrison wrote:
> > > > Isn't that what quiescent=yes is for? At least in masquerade mode
> > > > (what we use), when the weight is set to 0 ipvs continues to send
> > > > existing TCP sessions to that server. We weight a server to 0,
> > > > then watch to make sure all of the connections have closed, then
> > > > bring the service down.
> > >
> > > Just to clarify, quiescent=yes should work this way for all of LVS's
> > > forwarding methods.
> > >
> > It does not for SH (and DH I presume but I never tested that). Well,
> > not as expected at least. As I've been pointing out for well over a
> > year and just re-tested with the latest kernel.
> >
> > If the weight goes to 0 with SH and quiescent=yes set connections will
> > keep going to the old server as expected. Alas even when they expire
> > they will keep being sent there as will completely new connections if
> > the hash result assigns them to the weight 0 RS.
>
> I don't seem to be able to reproduce that problem.

Debian Sid install (2.6.29 bleeding edge kernel, but anything 2.6 and hand
rolled kernels are the same), Heartbeat with ldirectord and this config:
---
$ cat /etc/ha.d/conf/web
# Global Directives
checktimeout=10
checkinterval=2
autoreload=no
quiescent=yes

# Virtual Server for HTTP
virtual=203.216.90.88:80
# fallback=127.0.0.1:80
real=203.216.90.82:80 gate
real=203.216.90.83:80 gate
service=http
scheduler=sh
protocol=tcp
checktype=connect
emailalertfreq=600
emailalert=root
---

Shutting down Apache on the 2nd RS or manually setting its weight to 0 via
ipvsadm (for a planned maintenance) will result in the above scenario.
Only with quiescent=no and the resulting impact for maintenance things
work as expected.

> Do you have persistence set? If so, could you try setting
> /proc/sys/net/ipv4/vs/expire_quiescent_template to 1 ?
>
As you can see there is no persistence set in the config and back when I
reported this I played with all those flags. But here I go again, it is 0
by default and setting it to 1 does not change that behavior.

If there is anything else you would like me to do or try, please let me
know. Access to these or other test boxes could be arranged, too.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi [at] gol Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

chibi at gol

May 22, 2009, 2:08 AM

Post #11 of 13 (1930 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

Hello,

One more thing I just noticed (must be part of the newer kernel since I
never saw it before is that it states this in the kernel log when trying
to (falsely) route packets to the dead RS:
---
May 22 18:04:45 engtest02 kernel: [282126.815844] IPVS: SH: no destination available
---

Regards,

Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi [at] gol Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

May 25, 2009, 11:27 PM

Post #12 of 13 (1837 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

On Fri, May 22, 2009 at 06:08:18PM +0900, Christian Balzer wrote:
>
> Hello,
>
> One more thing I just noticed (must be part of the newer kernel since I
> never saw it before is that it states this in the kernel log when trying
> to (falsely) route packets to the dead RS:
> ---
> May 22 18:04:45 engtest02 kernel: [282126.815844] IPVS: SH: no destination available

Is this causing trouble?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

chibi at gol

May 26, 2009, 10:10 PM

Post #13 of 13 (1822 views)
Permalink

Re: [lvs-users] ldirectord feature patch - add abilility to signal system maintenance [In reply to]

Hello Simon,

On Tue, 26 May 2009 16:27:45 +1000 Simon Horman wrote:
> On Fri, May 22, 2009 at 06:08:18PM +0900, Christian Balzer wrote:
> >
> > Hello,
> >
> > One more thing I just noticed (must be part of the newer kernel since I
> > never saw it before is that it states this in the kernel log when
> > trying to (falsely) route packets to the dead RS:
> > ---
> > May 22 18:04:45 engtest02 kernel: [282126.815844] IPVS: SH: no
> > destination available
>
> Is this causing trouble?
>
Err, come again? It is the first time that I see the scheduler actually
logging that it is falling flat on it's face despite the fact that it very
much has a working realserver and thus a destination available.
It is indication/acknowledgment of the problems with SH and quiescent=yes
I pointed out earlier in this thread.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer NOC
chibi [at] gol Global OnLine Japan/Fusion Network Services
http://www.gol.com/
https://secure3.gol.com/mod-pl/ols/index.cgi/?intr_id=F-2ECXvzcr6656

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads