Mailing List Archive: Linux Virtual Server: Users

[lvs-users] Directors, single point of failure

Index | Next | Previous | View Threaded

msmart at smartsoftwareinc

Oct 29, 2007, 9:14 AM

Post #1 of 3 (557 views)
Permalink

[lvs-users] Directors, single point of failure

I have a production cluster running lvs-tun with realservers distributed
geographically in order to provide high availability. I have a director
and a backup, but they exist on the same local network. Last week, there
was a failure in the fiber line connecting my directors to the internet
and the whole cluster went down as a result...

I am trying to eliminate single points of failure in my cluster. I have
redundant fiber coming in, but that still leaves me with my directors
being in the same physical location (and as such, vulnerable to natural
disasters). Is there literature available (in any form whatsoever),
describing how to setup a system with multiple directors in different
geographic locations? The ideal situation would be one in which the
complete loss of a cluster location (including one or more directors)
would have little to no effect on the operation of the cluster as a whole...

I thought about setting up rr dns to point to different directors, but
was not sure if there would be issues with that approach, and am not
sure what would happen if one of those directors went down... would half
my requests get routed to a nonexistant director for the dns ttl period?

Anyway, thanks for any info you can point me to.

Matthew Smart
President
Smart Software Solutions Inc.
108 S Pierre St.
Pierre, SD 57501

Phone: (605) 280-0383
Skype: msmart13
Email: msmart [at] smartsoftwareinc

Dan Yocum wrote:
> Graeme Fowler wrote:
>
>> On Mon, 2007-10-29 at 09:13 -0500, Dan Yocum wrote:
>>
>>> It is possible to create a service certificate with a wildcard in the CN
>>> string.
>>>
>> It is indeed; however please note that there are several browsers which
>> don't accept wildcards, particularly embedded browsers or those in
>> mobile devices. They'll either throw an error/warning, or just not work
>> at all.
>>
>
> Ah, of course, you did specifically mention that. It didn't sink in to
> my little, full brain.
>
> Cheers,
> Dan
>
>

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Oct 29, 2007, 10:00 AM

Post #2 of 3 (543 views)
Permalink

Re: [lvs-users] Directors, single point of failure [In reply to]

On Mon, 29 Oct 2007, Matthew Smart wrote:

> I have a production cluster running lvs-tun with realservers distributed
> geographically in order to provide high availability. I have a director
> and a backup, but they exist on the same local network. Last week, there
> was a failure in the fiber line connecting my directors to the internet
> and the whole cluster went down as a result...

it's a problem and comes up occasionally here. I think it
may be discussed in the HOWTO (I should know, but I've
forgotten).

There is no cheap solution. If you have a two pairs of
directors in separate locations, then the data centers have
to cooperate (usually the same company) and they have to do
fast DNS updates.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Oct 29, 2007, 12:35 PM

Post #3 of 3 (537 views)
Permalink

Re: [lvs-users] Directors, single point of failure [In reply to]

On Mon, 2007-10-29 at 10:00 -0700, Joseph Mack NA3T wrote:
> There is no cheap solution. If you have a two pairs of
> directors in separate locations, then the data centers have
> to cooperate (usually the same company) and they have to do
> fast DNS updates.

I'd caution against using round-robin resource records in DNS, since
there's a significant number of providers out there who cache records
for longer than their TTL dictates. Once a given user on a given ISP has
one, if it disappears (in IP terms) they're stuck until their providers
caching nameserver expires the record.

As Joe says, there's no cheap way to do this.

One way to achieve some level of distributed resilience is to move the
connectivity endpoint back to your providers, by colocating the far end
routers yourself and then using their backhaul to your hosting
centre(s). This way you can put the directors out in the colo, keeping
the realservers back in your facility(ies).

Using a reasonable router and something like BGP, you have ultimate
control over what prefixes are where within your network - so you can
keep the VIPs at the far end, but logically keep them in the same
network(s) as the facility(ies).

This way, if a fibre goes down then you still have a director alive.

All that said - you say that the problem you experienced was:

"Last week, there was a failure in the fiber line connecting my
directors to the internet and the whole cluster went down as a
result..."

Why not have multiple connections to your directors from your redundant
connections?

You may have to give some more detail about your network topology to get
much further (noting that none of it's to do with LVS at all, it's just
good network design).

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads