Mailing List Archive: Linux Virtual Server: Users

[lvs-users] FW: Self Configuring High Availability LVS with Keepalived

Index | Next | Previous | View Threaded

hinkle at cipafilter

Oct 23, 2007, 1:15 PM

Post #1 of 7 (781 views)
Permalink

[lvs-users] FW: Self Configuring High Availability LVS with Keepalived

I'm trying to put together a self configuring high availability LVS. Each of my units is a proxy server. Each of them is identical, except a little configuration, and I intend them to come up, elect a director, and serve content.

I notice when I have a unit in BACKUP state, when I do "ipvsadm" I see all the lvs stuff is configured in the kernel.

Does this mean that the unit will try to do load balancing when a connection comes in instead of just ignoring it and allowing the proxy server's user space software to process it?

If so, Can I configure keepalived in such a way that the ipvsadm stuff isn't set up until a unit goes to MASTER? And if so, will the connection state information successfully synchronize?

David

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Oct 23, 2007, 5:34 PM

Post #2 of 7 (756 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

On Tue, 23 Oct 2007, David Hinkle wrote:

> I'm trying to put together a self configuring high
> availability LVS. Each of my units is a proxy server.
> Each of them is identical, except a little configuration,
> and I intend them to come up, elect a director, and serve
> content.

this was a highly hoped for goal in the early days of LVS.
The director isn't all that different to a realserver. When
a director fails out, one of the realservers should be able
to take over, presumably automatically. The service
listening on the VIP would have to be shut down on going
from realserver to director.

However one we got the two director failover working, people
lost interest. One working scheme was enough, at least for
the moment.

> I notice when I have a unit in BACKUP state, when I do
> "ipvsadm" I see all the lvs stuff is configured in the
> kernel.
>
> Does this mean that the unit will try to do load balancing
> when a connection comes in instead of just ignoring it and
> allowing the proxy server's user space software to process
> it?

usually a director in backup state doesn't have the VIP, so
there are no packets for it to ignore. If you sent it packet
for the VIP backup director would loadbalance them.

> If so, Can I configure keepalived in such a way that the
> ipvsadm stuff isn't set up until a unit goes to MASTER?

it doesn't take any time to setup ipvsadm, so yes.

> And if so, will the connection state information
> successfully synchronize?

once the demon starts and lets the partner demon in the
other director know where it is

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

hinkle at cipafilter

Oct 23, 2007, 8:53 PM

Post #3 of 7 (759 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

Eventually I did get this working today. I keep the VIP on every lo in the cluster. Every machine in the cluster starts as a backup director. One of them will get elected as Master director by keepalived and all the rest will receive syncronization broadcasts from the master director.

The trick was, I used netfilter Marks to determine what traffic to load balance, and I configured keepalived to add the netfilter rules to iptables when it becomes a master and remove them when it's a backup.

This way, LVS on the backups does not try to load balance incoming connections a second time, and they can just act as real servers.

This gives me a cluster where all machines are peers. One of them just happens to be the director at any given time.

I'm going to use this to provide 1 click clustering for our CIPAFilter products. They already have the ability to detect each other via broadcasts, so after a little more glue tommorow, a customer with load problems will be able to slap a few more cipafilters into their rack, click one checkbox, and have a fully load balanced high availability cluster.

Great work on this stuff guys. Honestly, I had no idea it was so powerful or I would have implemented it years ago.

David

-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver on behalf of Joseph Mack NA3T
Sent: Tue 10/23/2007 7:34 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived

On Tue, 23 Oct 2007, David Hinkle wrote:

> I'm trying to put together a self configuring high
> availability LVS. Each of my units is a proxy server.
> Each of them is identical, except a little configuration,
> and I intend them to come up, elect a director, and serve
> content.

this was a highly hoped for goal in the early days of LVS.
The director isn't all that different to a realserver. When
a director fails out, one of the realservers should be able
to take over, presumably automatically. The service
listening on the VIP would have to be shut down on going
from realserver to director.

However one we got the two director failover working, people
lost interest. One working scheme was enough, at least for
the moment.

> I notice when I have a unit in BACKUP state, when I do
> "ipvsadm" I see all the lvs stuff is configured in the
> kernel.
>
> Does this mean that the unit will try to do load balancing
> when a connection comes in instead of just ignoring it and
> allowing the proxy server's user space software to process
> it?

usually a director in backup state doesn't have the VIP, so
there are no packets for it to ignore. If you sent it packet
for the VIP backup director would loadbalance them.

> If so, Can I configure keepalived in such a way that the
> ipvsadm stuff isn't set up until a unit goes to MASTER?

it doesn't take any time to setup ipvsadm, so yes.

> And if so, will the connection state information
> successfully synchronize?

once the demon starts and lets the partner demon in the
other director know where it is

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

hinkle at cipafilter

Oct 23, 2007, 8:53 PM

Post #4 of 7 (759 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

Oh, just a quick suggestion. This hack would be completely unnecessary if you could simply configure a backup LVS server not to try and load balance. I just want a backup to sit there and collect data, I don't want it to try and load balance anything until I promote it! :)

David

-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver on behalf of Joseph Mack NA3T
Sent: Tue 10/23/2007 7:34 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived

On Tue, 23 Oct 2007, David Hinkle wrote:

> I'm trying to put together a self configuring high
> availability LVS. Each of my units is a proxy server.
> Each of them is identical, except a little configuration,
> and I intend them to come up, elect a director, and serve
> content.

this was a highly hoped for goal in the early days of LVS.
The director isn't all that different to a realserver. When
a director fails out, one of the realservers should be able
to take over, presumably automatically. The service
listening on the VIP would have to be shut down on going
from realserver to director.

However one we got the two director failover working, people
lost interest. One working scheme was enough, at least for
the moment.

> I notice when I have a unit in BACKUP state, when I do
> "ipvsadm" I see all the lvs stuff is configured in the
> kernel.
>
> Does this mean that the unit will try to do load balancing
> when a connection comes in instead of just ignoring it and
> allowing the proxy server's user space software to process
> it?

usually a director in backup state doesn't have the VIP, so
there are no packets for it to ignore. If you sent it packet
for the VIP backup director would loadbalance them.

> If so, Can I configure keepalived in such a way that the
> ipvsadm stuff isn't set up until a unit goes to MASTER?

it doesn't take any time to setup ipvsadm, so yes.

> And if so, will the connection state information
> successfully synchronize?

once the demon starts and lets the partner demon in the
other director know where it is

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sameer.garg at gmail

Oct 23, 2007, 11:17 PM

Post #5 of 7 (754 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

Just a thought for people using heartbeat and ldirectord. Heartbeat
allows for custom bash scripts. So one could use a custom script to
shutdown the service and bring up the VIP on the chosen realserver.

Sameer

On 10/24/07, David Hinkle <hinkle [at] cipafilter> wrote:
> Oh, just a quick suggestion. This hack would be completely unnecessary if you could simply configure a backup LVS server not to try and load balance. I just want a backup to sit there and collect data, I don't want it to try and load balance anything until I promote it! :)
>
> David
>
>
> -----Original Message-----
> From: lvs-users-bounces [at] linuxvirtualserver on behalf of Joseph Mack NA3T
> Sent: Tue 10/23/2007 7:34 PM
> To: LinuxVirtualServer.org users mailing list.
> Subject: Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived
>
> On Tue, 23 Oct 2007, David Hinkle wrote:
>
> > I'm trying to put together a self configuring high
> > availability LVS. Each of my units is a proxy server.
> > Each of them is identical, except a little configuration,
> > and I intend them to come up, elect a director, and serve
> > content.
>
> this was a highly hoped for goal in the early days of LVS.
> The director isn't all that different to a realserver. When
> a director fails out, one of the realservers should be able
> to take over, presumably automatically. The service
> listening on the VIP would have to be shut down on going
> from realserver to director.
>
> However one we got the two director failover working, people
> lost interest. One working scheme was enough, at least for
> the moment.
>
> > I notice when I have a unit in BACKUP state, when I do
> > "ipvsadm" I see all the lvs stuff is configured in the
> > kernel.
> >
> > Does this mean that the unit will try to do load balancing
> > when a connection comes in instead of just ignoring it and
> > allowing the proxy server's user space software to process
> > it?
>
> usually a director in backup state doesn't have the VIP, so
> there are no packets for it to ignore. If you sent it packet
> for the VIP backup director would loadbalance them.
>
> > If so, Can I configure keepalived in such a way that the
> > ipvsadm stuff isn't set up until a unit goes to MASTER?
>
> it doesn't take any time to setup ipvsadm, so yes.
>
> > And if so, will the connection state information
> > successfully synchronize?
>
> once the demon starts and lets the partner demon in the
> other director know where it is
>
> Joe
>
> --
> Joseph Mack NA3T EME(B,D), FM05lw North Carolina
> jmack (at) wm7d (dot) net - azimuthal equidistant map
> generator at http://www.wm7d.net/azproj.shtml
> Homepage http://www.austintek.com/ It's GNU/Linux!
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Oct 24, 2007, 6:21 AM

Post #6 of 7 (752 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

On Tue, 23 Oct 2007, David Hinkle wrote:

> Oh, just a quick suggestion. This hack would be
> completely unnecessary if you could simply configure a
> backup LVS server not to try and load balance.

do you know that in the current way of doing director
failover that the backup director is getting the ipvsadm
state table from the active director, so that it when the
backup becomes the active, that it knows the state of the
lvs controlled connections?

If you know that already, what's missing from LVS that you
need?

> I just want a backup to sit there and collect data, I
> don't want it to try and load balance anything until I
> promote it! :)

It doesn't loadbalance till you send it packets.

It seems that you and I have a different idea about what's
happening here and I don't know what you idea is. Can you
straighten me out?

Thanks

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

hinkle at cipafilter

Oct 24, 2007, 8:38 AM

Post #7 of 7 (753 views)
Permalink

Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived [In reply to]

I want my backup directors to be real servers, and I've succeeded, so there's nothing I "need" at this point.

The problem with it routing when it's a backup is that it mungs up the incomming connections to the realserver running locally when the Master DR sends connections to it.

I solved my problem by using netfilter marking to direct packets to be load balanced, and only marking packets when the unit is master, thereby working around the issue.

My recomendation is that the backups not try to load balance until they're switched to master by the userspace stuff.

This is an issue for me because all my units are "peers", and I want to the master director to be able to be any one of them so I can do nifty stuff like rolling firmware upgrades, with 0 downtime, etc. I'm sure my needs are rather specialized, and even then I was able to get them met, just by jumping through a few hoops, so great job!

David

-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver on behalf of Joseph Mack NA3T
Sent: Wed 10/24/2007 8:21 AM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] FW: Self Configuring High Availability LVS with Keepalived

On Tue, 23 Oct 2007, David Hinkle wrote:

> Oh, just a quick suggestion. This hack would be
> completely unnecessary if you could simply configure a
> backup LVS server not to try and load balance.

do you know that in the current way of doing director
failover that the backup director is getting the ipvsadm
state table from the active director, so that it when the
backup becomes the active, that it knows the state of the
lvs controlled connections?

If you know that already, what's missing from LVS that you
need?

> I just want a backup to sit there and collect data, I
> don't want it to try and load balance anything until I
> promote it! :)

It doesn't loadbalance till you send it packets.

It seems that you and I have a different idea about what's
happening here and I don't know what you idea is. Can you
straighten me out?

Thanks

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads