Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS with Weighted Least-Connection Scheduling

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


sebykrueger at gmail

Feb 20, 2008, 6:15 PM

Post #1 of 7 (424 views)
Permalink
[lvs-users] LVS with Weighted Least-Connection Scheduling

Hi everyone,

I'm trying to balance the load on 3 servers. The way this is set up is as
that LVS presents three virtual servers on the following ports:

10.32.30.125:18007
10.32.30.125:18008
10.32.30.125:18009

These I've set up in the ldirectord.cf like this:

virtual=10.32.30.125:18007
real=192.168.0.10:18007 masq 10
real=192.168.0.20:18007 masq 5
real=192.168.0.30:18007 masq 5
checktype=on
virtualhost=eaivcon3.kdc.ird.govt.nz
scheduler=wlc
persistent=1
protocol=tcp
checkport=18001
service=http
request="/PingWebClient/pingApp.jsp"
receive="REPOUT"
virtual=10.32.30.125:18008
real=192.168.0.10:18007 masq 5
real=192.168.0.20:18007 masq 10
real=192.168.0.30:18007 masq 5
checktype=on
virtualhost=eaivcon3.kdc.ird.govt.nz
scheduler=wlc
persistent=1
protocol=tcp
checkport=18001
service=http
request="/PingWebClient/pingApp.jsp"
receive="REPOUT"
virtual=10.32.30.125:18009
real=192.168.0.10:18007 masq 5
real=192.168.0.20:18007 masq 5
real=192.168.0.30:18007 masq 10
checktype=on
virtualhost=eaivcon3.kdc.ird.govt.nz
scheduler=wlc
persistent=1
protocol=tcp
service=http
checkport=18001
request="/PingWebClient/pingApp.jsp"
receive="REPOUT"


Notice how I have given each virtual server a different weighting for the
real servers. My aim was that if each virtual server has a single distinct
client connected, then they would all be balanced to different real servers.
Based on the weightings specified above.

However, unfortunately traffic is assigned to an arbitrary real server for
each virtual server if there are no other connections already established.

Has anyone found a way around this?

Regards, Sebastian.
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Feb 20, 2008, 6:38 PM

Post #2 of 7 (410 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

On Thu, 21 Feb 2008, Sebastian Krueger wrote:

> persistent=1

> Notice how I have given each virtual server a different weighting for the
> real servers. My aim was that if each virtual server has a single distinct
> client connected, then they would all be balanced to different real servers.
> Based on the weightings specified above.

if persistent=1 means persistence on, you may see something
different.

> However, unfortunately traffic is assigned to an arbitrary real server for
> each virtual server if there are no other connections already established.

the virtual services are independant. ipvsadm when assigning
the realserver for one virtual service doesn't care that the
same realserver also has 100000 connections from another
virtual service

> Has anyone found a way around this?

If I've interpreted your question correctly, the behaviour
is part of the design spec.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Feb 20, 2008, 6:43 PM

Post #3 of 7 (409 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

On Wed, 20 Feb 2008, Joseph Mack NA3T wrote:

> If I've interpreted your question correctly, the behaviour
> is part of the design spec.

if you wanted all 3 virtual services to be scheduled as one
service, you could make them one service with fwmark

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


gwaugh at scu

Feb 20, 2008, 6:50 PM

Post #4 of 7 (408 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

Hi Sebastian,

Sebastian Krueger wrote:
> Hi everyone,

<snip>

> Notice how I have given each virtual server a different weighting for the
> real servers. My aim was that if each virtual server has a single distinct
> client connected, then they would all be balanced to different real servers.
> Based on the weightings specified above.

Are you saying that there will only ever be 3 distinct clients
connected, and you want each of them to be balanced to a separate real
server?

If so, you must be pointing each client at their own virtual service
address, so you may as well do away with LVS altogether and just point
them at their own real server directly.

If not, just use one virtual service, load-balanced between your three
real servers equally (i.e. each real server has equal weight), but
consider using a persistence value, so that each client will 'stay' with
a real server for a certain amount of time... like this:

virtual=10.32.30.125:18007
real=192.168.0.10:18007 masq 1
real=192.168.0.20:18007 masq 1
real=192.168.0.30:18007 masq 1
checktype=on
virtualhost=eaivcon3.kdc.ird.govt.nz
scheduler=wlc
persistent=600
protocol=tcp
checkport=18001
service=http
request="/PingWebClient/pingApp.jsp"
receive="REPOUT"

... or then again, perhaps I am missing what you are getting at entirely 8-)

>
> However, unfortunately traffic is assigned to an arbitrary real server for
> each virtual server if there are no other connections already established.

Yes - presumably the LVS director is load-balancing requests between the
three real servers. The weights probably only come into it once there
are a few connections happening.

Regards,
Guy.

>
> Has anyone found a way around this?
>
> Regards, Sebastian.
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


sebykrueger at gmail

Feb 20, 2008, 7:48 PM

Post #5 of 7 (409 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

Joe,

I checked out persistent=1 and from the man page of ldirectord I find this:

----------
persistent = n

Number of seconds for persistent client connections.
----------

So it wouldn't be that I think.

fwmark sounds interesting, I will google about that one in a moment to find
out.

Guy,

>>
>> However, unfortunately traffic is assigned to an arbitrary real server
for
>> each virtual server if there are no other connections already
established.
>
>Yes - presumably the LVS director is load-balancing requests between the
>three real servers. The weights probably only come into it once there
>are a few connections happening.

I have a feeling you might be right. Maybe I'll end up having to change the
scheduling algorithm myself to include the weights if no clients are
connected.

What I'm trying to achieve is to load-balance Java Message Service (JMS)
connections.

So we have multiple JMS clients and multiple JMS servers. LVS does the
load-balancing
based on IP address affinity, so in order to load-balance a single JMS
client across many
JMS virtual servers the solution was to create multiple virtual servers and
have the
JMS client round-robin over all the available JMS virtual servers.

Currently these are set to 3, but the LVS configuration could easily be
extended to

virtual=10.32.30.125:18007
real=192.168.0.10:18007 masq 10
real=192.168.0.20:18007 masq 5
real=192.168.0.30:18007 masq 5
real=192.168.0.40:18007 masq 5
real=192.168.0.50:18007 masq 5
real=192.168.0.60:18007 masq 5
virtual=10.32.30.125:18008
real=192.168.0.10:18007 masq 5
real=192.168.0.20:18007 masq 10
real=192.168.0.30:18007 masq 5
real=192.168.0.40:18007 masq 5
real=192.168.0.50:18007 masq 5
real=192.168.0.60:18007 masq 5

So now a single JMS client can load-balance itself across 2 virtual servers
and many
JMS clients are load-balanced across all real-servers so the over-all load
is
nicely spread across all real servers.

So let's say a single JMS client connects to both virtual JMS servers.
If no other clients are connected, he will be scheduled to
the real server on 192.168.0.10 on both virtual servers. This is because LVS
chooses the first real server regardless of the weighting if there are no
existing
connections.

However, since each received JMS message means a huge amount of processing,
it is important that each request goes to different real servers (as much
as possible).

Is there anyone out there that has ever tried to use LVS for JMS
connections?

Thanks for your help!

Regards, Sebastian.

On Thu, Feb 21, 2008 at 3:50 PM, Guy Waugh <gwaugh [at] scu> wrote:

> Hi Sebastian,
>
> Sebastian Krueger wrote:
> > Hi everyone,
>
> <snip>
>
> > Notice how I have given each virtual server a different weighting for
> the
> > real servers. My aim was that if each virtual server has a single
> distinct
> > client connected, then they would all be balanced to different real
> servers.
> > Based on the weightings specified above.
>
> Are you saying that there will only ever be 3 distinct clients
> connected, and you want each of them to be balanced to a separate real
> server?
>
> If so, you must be pointing each client at their own virtual service
> address, so you may as well do away with LVS altogether and just point
> them at their own real server directly.
>
> If not, just use one virtual service, load-balanced between your three
> real servers equally (i.e. each real server has equal weight), but
> consider using a persistence value, so that each client will 'stay' with
> a real server for a certain amount of time... like this:
>
> virtual=10.32.30.125:18007
> real=192.168.0.10:18007 masq 1
> real=192.168.0.20:18007 masq 1
> real=192.168.0.30:18007 masq 1
> checktype=on
> virtualhost=eaivcon3.kdc.ird.govt.nz
> scheduler=wlc
> persistent=600
> protocol=tcp
> checkport=18001
> service=http
> request="/PingWebClient/pingApp.jsp"
> receive="REPOUT"
>
> ... or then again, perhaps I am missing what you are getting at entirely
> 8-)
>
> >
> > However, unfortunately traffic is assigned to an arbitrary real server
> for
> > each virtual server if there are no other connections already
> established.
>
> Yes - presumably the LVS director is load-balancing requests between the
> three real servers. The weights probably only come into it once there
> are a few connections happening.
>
> Regards,
> Guy.
>
> >
> > Has anyone found a way around this?
> >
> > Regards, Sebastian.
> > _______________________________________________
> > LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> > Send requests to lvs-users-request [at] LinuxVirtualServer
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


gwaugh at scu

Feb 20, 2008, 9:09 PM

Post #6 of 7 (409 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

Sebastian Krueger wrote:
> Joe,
>
> I checked out persistent=1 and from the man page of ldirectord I find this:
>
> ----------
> persistent = n
>
> Number of seconds for persistent client connections.
> ----------
>
> So it wouldn't be that I think.
>
> fwmark sounds interesting, I will google about that one in a moment to find
> out.
>
> Guy,
>
>>> However, unfortunately traffic is assigned to an arbitrary real server
> for
>>> each virtual server if there are no other connections already
> established.
>> Yes - presumably the LVS director is load-balancing requests between the
>> three real servers. The weights probably only come into it once there
>> are a few connections happening.
>
> I have a feeling you might be right. Maybe I'll end up having to change the
> scheduling algorithm myself to include the weights if no clients are
> connected.
>
> What I'm trying to achieve is to load-balance Java Message Service (JMS)
> connections.
>
> So we have multiple JMS clients and multiple JMS servers. LVS does the
> load-balancing
> based on IP address affinity, so in order to load-balance a single JMS
> client across many
> JMS virtual servers the solution was to create multiple virtual servers and
> have the
> JMS client round-robin over all the available JMS virtual servers.
>
> Currently these are set to 3, but the LVS configuration could easily be
> extended to
>
> virtual=10.32.30.125:18007
> real=192.168.0.10:18007 masq 10
> real=192.168.0.20:18007 masq 5
> real=192.168.0.30:18007 masq 5
> real=192.168.0.40:18007 masq 5
> real=192.168.0.50:18007 masq 5
> real=192.168.0.60:18007 masq 5
> virtual=10.32.30.125:18008
> real=192.168.0.10:18007 masq 5
> real=192.168.0.20:18007 masq 10
> real=192.168.0.30:18007 masq 5
> real=192.168.0.40:18007 masq 5
> real=192.168.0.50:18007 masq 5
> real=192.168.0.60:18007 masq 5
>
> So now a single JMS client can load-balance itself across 2 virtual servers
> and many
> JMS clients are load-balanced across all real-servers so the over-all load
> is
> nicely spread across all real servers.
>
> So let's say a single JMS client connects to both virtual JMS servers.
> If no other clients are connected, he will be scheduled to
> the real server on 192.168.0.10 on both virtual servers. This is because LVS
> chooses the first real server regardless of the weighting if there are no
> existing
> connections.
>
> However, since each received JMS message means a huge amount of processing,
> it is important that each request goes to different real servers (as much
> as possible).
>
> Is there anyone out there that has ever tried to use LVS for JMS
> connections?

Hi again,

I don't know anything about JMS, but if it is possible for one JMS
client to be configured to use more than one JMS server at the same
time, then I can't help thinking that you'll only really need the
three-virtual-service-address model if you have around the same number
of clients as real servers. With, say, twice the number of clients as
real servers, all real servers are going to have an average of two
clients anyway (assuming the weights for all of them are equal), so
splitting parts of each client's request between real servers won't
achieve anything.

I'm only familiar with using LVS in a way such that each client is
load-balanced to one real server... perhaps you would need Layer 7 load
balancing to load-balance according to certain attributes of the request.

Alternatively, using a persistence value of zero (if this is indeed
possible), so that each request is load-balanced afresh, might be what
you're wanting.

Cheers,
Guy.

>
> Thanks for your help!
>
> Regards, Sebastian.

<snip>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Feb 21, 2008, 7:29 AM

Post #7 of 7 (404 views)
Permalink
Re: [lvs-users] LVS with Weighted Least-Connection Scheduling [In reply to]

On Thu, 21 Feb 2008, Sebastian Krueger wrote:

> Joe,
>
> I checked out persistent=1 and from the man page of ldirectord I find this:
>
> ----------
> persistent = n
>
> Number of seconds for persistent client connections.
> ----------

didn't know what that meant. persistence of 1 sec is not
going to do anything useful though.

Have you looked at the -SH scheduler to acheive
client-realserver affinity? It has less problems than
persistence.

> fwmark sounds interesting, I will google about that one in a moment to find
> out.

if we know about it, it will be in the HOWTO.

> What I'm trying to achieve is to load-balance Java Message Service (JMS)
> connections.

I don't know anything about it. I'll assume it's a service
with relatively long connection time and requires heavy
compute service on the realserver.

I don't know why you have three apparenlty identical
services on consecutive ports. It seems only to make the
setup more complicated.

>
> So we have multiple JMS clients and multiple JMS servers. LVS does the
> load-balancing
> based on IP address affinity, so in order to load-balance a single JMS
> client across many
> JMS virtual servers the solution was to create multiple virtual servers and
> have the
> JMS client round-robin over all the available JMS virtual servers.
>
> Currently these are set to 3, but the LVS configuration could easily be
> extended to
>
> virtual=10.32.30.125:18007
> real=192.168.0.10:18007 masq 10
> real=192.168.0.20:18007 masq 5
> real=192.168.0.30:18007 masq 5
> real=192.168.0.40:18007 masq 5
> real=192.168.0.50:18007 masq 5
> real=192.168.0.60:18007 masq 5
> virtual=10.32.30.125:18008
> real=192.168.0.10:18007 masq 5
> real=192.168.0.20:18007 masq 10
> real=192.168.0.30:18007 masq 5
> real=192.168.0.40:18007 masq 5
> real=192.168.0.50:18007 masq 5
> real=192.168.0.60:18007 masq 5

is the realserver listening on the same port for both
virtualservices?

> So let's say a single JMS client connects to both virtual JMS servers.
> If no other clients are connected, he will be scheduled to
> the real server on 192.168.0.10 on both virtual servers. This is because LVS
> chooses the first real server regardless of the weighting if there are no
> existing
> connections.
>
> However, since each received JMS message means a huge amount of processing,
> it is important that each request goes to different real servers (as much
> as possible).

LVS statistics are designed for large numbers of
connections. Originally LVS was used for http, where the
connections are short and there are thousands of connections
open. Presumably connections for small numbers of
connections is deterministic, but I don't think you have
much control over it.

> Is there anyone out there that has ever tried to use LVS for JMS
> connections?

can you edit your posts. Nothing below here is relevant but
it's being sent to 3 archivers and 1000 subscribers.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.