Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] One-to-many dns load balancing and HA/HR questions

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


erik-lvs at arpa

Mar 21, 2011, 5:50 PM

Post #1 of 5 (669 views)
Permalink
[lvs-users] One-to-many dns load balancing and HA/HR questions

I plan to deploy a lab environment to start testing LVS as a load
balancer in front of a group of what could be called nameservers. These
nameservers are actually serving telephone call routing, filtering, and
translation data using UDP dns-style queries, and in our production
environment normally serve 500-2000 queries per second, each.

The "clients" initiating these queries are ACME session border
controllers and various other VOIP/SIP processing equipment. A failure
of the involved systems to pass a lookup to the servers, process the
lookup, return a response, and route it back to the client is considered
critical as it means a call gets dropped or is left with "dead air".
Best case, the call gets delayed by a few seconds as a request times out
and (hopefully) gets processed by a device that is able to respond to
the retransmitted query.

I'm aware of the benefit to lowering the UDP session timeout to 15
seconds for high-volume DNS load balancing and plan to do this, but I
was wondering if LVS/IPVS incorporates methods to guarantee delivery of
a UDP request packet to a server that's able to respond to it, no matter
what.

In other words, if a DNS request comes into the VIP on the load
balancer, the load balancer forwards it (either via routing or nat) to a
"real server", but that real server is unable to correctly receive that
packet or process the query it contains for any reason, be it a dropped
packet on the wire, intermittent CPU saturation, a missed interrupt,
etc, then it would be desirable for the load balancer to detect that a
response has not been sent back to the client from the realserver and
basically re-send the same packet (same payload) to another real server
in the cluster. The typical time it takes one of these servers to
respond is usually less than 50ms, but may be as high as 100ms. If
200ms has passed after a request and the chosen server hasn't responded
yet, retransmit a copy of the original request packet to a new server
without the requesting client realizing there was a timeout.

Is this possible?

When there are 10,000 requests being processed per second, dropping even
one packet per 100,000 is disastrous for our stats.



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


netdev at bof

Mar 22, 2011, 12:42 AM

Post #2 of 5 (648 views)
Permalink
Re: [lvs-users] One-to-many dns load balancing and HA/HR questions [In reply to]

On Mon, 2011-03-21 at 17:50 -0700, Erik Schorr wrote:

[.SNIP wish of identifying individual UDP transaction failure and
reassignment to a different real server]

> Is this possible?

Not without investing in the implementation of an extension of the
kernel part of IPVS.

No part of IPVS cares about / tries to do something regarding
reassignment of individual "failed" flows to different real servers.

It is up to a userlevel health checking application (keepalived,
ldirectord) to test and disable real servers that fail.

The kernel part just distributes new flows, according to the chosen
scheduler, to any of the non-weight-0 real servers configured, and
routes packets of known flows to the same real server as chosen
initially.

What you desire, could work in a NAT or TUN mode, but would need roughly
these new features:
A) a configuration variable, per virtual service, indicating that more
elaborate processing is desired, and in which time interval a reply
should be received.
B) keeping a copy of the data (UDP packeet, TCP SYN) sent initially to a
real server, the copy hanging off the IPVS connection (flow) structure.
C) put such new flows on a tight timeout configured by A)
D) when a reply packet is received and its flow identified (which
already must happen for e.g. NAT mode to work), mark the flow as OK and
remove it from the tight timeout schedule
E) when the tight timeout expires, rerun the scheduler selection,
excluding the initially selected real server (*), and send the
remembered copy of the datagram / TCP SYN to the newly selected real
server.
*) should one such failure set the weight of the failing real server to
0? Or decrease its weight? Or do nothing like that? The real server
might work almost perfectly, only having dropped somehow that single
datagram.

Further consideration might be given to the desired behaviour when
microseconds after the E) reassignment decision, the first real server
response is received, because it just sat in some queue-in-between for a
bit longer than anticipated.

best regards
Patrick



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Mar 22, 2011, 7:13 AM

Post #3 of 5 (657 views)
Permalink
Re: [lvs-users] One-to-many dns load balancing and HA/HR questions [In reply to]

On Mon, 21 Mar 2011, Erik Schorr wrote:

> I plan to deploy a lab environment to start testing LVS as a load
> balancer in front of a group of what could be called nameservers.

Regular (Bind) type DNS has its own loadbalancing. While you
can also put Bind behind an LVS director, it's not clear
that you get much by doing so. Maybe if you already have
an LVS and Bind is part of the installation, then you have
everything for your installation in one place.

Do your phone nameservers have their own loadbalancing?

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Mar 22, 2011, 7:46 AM

Post #4 of 5 (637 views)
Permalink
Re: [lvs-users] One-to-many dns load balancing and HA/HR questions [In reply to]

On Mon, 2011-03-21 at 17:50 -0700, Erik Schorr wrote:
...
> In other words, if a DNS request comes into the VIP on the load
> balancer, the load balancer forwards it (either via routing or nat) to a
> "real server", but that real server is unable to correctly receive that
> packet or process the query it contains for any reason, be it a dropped
> packet on the wire, intermittent CPU saturation, a missed interrupt,
> etc, then it would be desirable for the load balancer to detect that a
> response has not been sent back to the client from the realserver and
> basically re-send the same packet (same payload) to another real server
> in the cluster. The typical time it takes one of these servers to
> respond is usually less than 50ms, but may be as high as 100ms. If
> 200ms has passed after a request and the chosen server hasn't responded
> yet, retransmit a copy of the original request packet to a new server
> without the requesting client realizing there was a timeout.

So... that would be an application proxy, really. LVS/IPVS is an
"intelligent" (I use the word carefully!) router with rules dictating
where the traffic is sent. Once a packet is sent, that's it - there's no
state kept regarding content (ie. whether a connection was successful or
not) other than in the TCP sense, where the connection state is tracked
but *not* monitored. As with other routing devices, the application
layer is responsible for that bit of cleverness, and the routers are
below that layer.

Even with something like ldirectord, keepalived or mon monitoring your
realservers and changing the IPVS tables accordingly you wouldn't get
the guarantee you're after as your latency is so low.

You mention "UDP dns-style queries"; if this really is DNS then I would
consider building something different using a forwarding DNS server (or
more than one) which can then do the retries as necessary.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


erik-lvs at arpa

Mar 22, 2011, 10:54 AM

Post #5 of 5 (627 views)
Permalink
Re: [lvs-users] One-to-many dns load balancing and HA/HR questions [In reply to]

On 3/22/2011 12:42 AM, Patrick Schaaf wrote:
> On Mon, 2011-03-21 at 17:50 -0700, Erik Schorr wrote:
>
> [.SNIP wish of identifying individual UDP transaction failure and
> reassignment to a different real server]
>
>> Is this possible?
>
> Not without investing in the implementation of an extension of the
> kernel part of IPVS.
>
> No part of IPVS cares about / tries to do something regarding
> reassignment of individual "failed" flows to different real servers.
>
> It is up to a userlevel health checking application (keepalived,
> ldirectord) to test and disable real servers that fail.
Understood. It's not necessarily mitigation of failure, but enforcement
of best-effort forwarding within a deadline.

> The kernel part just distributes new flows, according to the chosen
> scheduler, to any of the non-weight-0 real servers configured, and
> routes packets of known flows to the same real server as chosen
> initially.
>
> What you desire, could work in a NAT or TUN mode, but would need roughly
> these new features:
> A) a configuration variable, per virtual service, indicating that more
> elaborate processing is desired, and in which time interval a reply
> should be received.
> B) keeping a copy of the data (UDP packeet, TCP SYN) sent initially to a
> real server, the copy hanging off the IPVS connection (flow) structure.
> C) put such new flows on a tight timeout configured by A)
> D) when a reply packet is received and its flow identified (which
> already must happen for e.g. NAT mode to work), mark the flow as OK and
> remove it from the tight timeout schedule
> E) when the tight timeout expires, rerun the scheduler selection,
> excluding the initially selected real server (*), and send the
> remembered copy of the datagram / TCP SYN to the newly selected real
> server.
> *) should one such failure set the weight of the failing real server to
> 0? Or decrease its weight? Or do nothing like that? The real server
> might work almost perfectly, only having dropped somehow that single
> datagram.
This is pretty much dead-on. For this last part, I think a configurable
threshold of "handoff-misses per time period" must be exceeded before a
real server's weight is reduced. One hand-off failure per 10 seconds,
perhaps, would decrease the weight by a percentage. Of course, if a
monitor detects a hard failure of a real server/service, then set the
weight to 0.

> Further consideration might be given to the desired behaviour when
> microseconds after the E) reassignment decision, the first real server
> response is received, because it just sat in some queue-in-between for a
> bit longer than anticipated.
In this case, I believe it would be fine for the load balancer to simply
drop the late reply.

Has anyone else encountered a situation with these sorts of
requirements? Load balancing and service monitoring is great, but
offering guaranteed connection-level reliability and deadline
enforcement are things I haven't seen offered except in very expensive
commercial systems. It would be interesting to know how many other
people might benefit from such features.

>
> best regards
> Patrick


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.