Mailing List Archive: Linux Virtual Server: Users

[lvs-users] Unbalanced Real Servers

Index | Next | Previous | View Threaded

goetz.rieger at gmx

Apr 29, 2009, 6:05 AM

Post #1 of 6 (1080 views)
Permalink

[lvs-users] Unbalanced Real Servers

Hi all,

I've set up a three-node (Apache 2.2/RHEL 5.3) web cluster with an LVS-LB (CentOS 5.2) in front and NFS storage in the back. I'm using keepalived on the LB. The hardware is identical and everything Apache (content and web server) lives on two NFS shares on a fourth node.

I'm doing a load test now using The Grinder with four load test clients. The problem I'm facing is an extremely uneven distribution of ActiveConn on the real servers:

[root [at] lvst ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 194.xxx.xxx.xxx:80 rr
-> 192.168.xxx.xxx:80 Masq 1 137 421
-> 192.168.xxx.xxx:80 Masq 1 203 509
-> 192.168.xxx.xxx:80 Masq 1 0 630

Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 194.xxx.xxx.xxx:80 rr
-> 192.168.xxx.xxx:80 Masq 1 232 424
-> 192.168.xxx.xxx:80 Masq 1 5 418
-> 192.168.xxx.xxx:80 Masq 1 3 419

The configuration is as easy as it gets:

[root [at] lvst ~]# cat lvs.conf
-A -t vs:http -s rr
-a -t vs:http -r node1:http -m -w 1
-a -t vs:http -r node2:http -m -w 1
-a -t vs:http -r node3:http -m -w 1

So persistence is disabled. The problem is of course the web servers are getting very uneven load, one is doing all the work while the others are idling around...

I'm now testing with the ll scheduler and now ActiveConn are exactly balanced. Is there any reason to not use ll? I've seen Willy's statement about ll not being suitable for a web server.

It might just be me lacking in-depth understanding but I would be grateful if somebody could give me a hand. I've already tried to digest all the How-To information.

Cheers,
Goetz

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Apr 29, 2009, 8:23 AM

Post #2 of 6 (1040 views)
Permalink

Re: [lvs-users] Unbalanced Real Servers [In reply to]

On Wed, 2009-04-29 at 15:05 +0200, "Götz Rieger" wrote:
> I'm doing a load test now using The Grinder with four load test clients. The problem I'm facing is an extremely uneven distribution of ActiveConn on the real servers:

OK...

> [root [at] lvst ~]# ipvsadm -L -n
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 194.xxx.xxx.xxx:80 rr
> -> 192.168.xxx.xxx:80 Masq 1 137 421
> -> 192.168.xxx.xxx:80 Masq 1 203 509
> -> 192.168.xxx.xxx:80 Masq 1 0 630
>
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 194.xxx.xxx.xxx:80 rr
> -> 192.168.xxx.xxx:80 Masq 1 232 424
> -> 192.168.xxx.xxx:80 Masq 1 5 418
> -> 192.168.xxx.xxx:80 Masq 1 3 419

A larger number of ActiveConns indicates a machine which is slower to
respond (that is, these connections have not reached TIME_WAIT or
FIN_WAIT), or a machine which has KeepAlive switched on in Apache.

Of more significance to you during testing would be:

ipvsadm -L -n --stats
ipvsadm -L -n --rate

If you use the "watch" utility you can watch these in real time:

watch -n1 -- "ipvsadm -L -n --stats; echo; echo; ipvsadm -L -n --rate"

BTW I presume you meant "lc" scheduler rather than "ll"?

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

thomas at pedoussaut

Apr 29, 2009, 8:32 AM

Post #3 of 6 (1025 views)
Permalink

Re: [lvs-users] Unbalanced Real Servers [In reply to]

Götz Rieger wrote:
> Hi all,
>
> I've set up a three-node (Apache 2.2/RHEL 5.3) web cluster with an LVS-LB (CentOS 5.2) in front and NFS storage in the back. I'm using keepalived on the LB. The hardware is identical and everything Apache (content and web server) lives on two NFS shares on a fourth node.
>
> I'm doing a load test now using The Grinder with four load test clients. The problem I'm facing is an extremely uneven distribution of ActiveConn on the real servers:
>
> [snip]

>
> The configuration is as easy as it gets:
>
> [root [at] lvst ~]# cat lvs.conf
> -A -t vs:http -s rr
> -a -t vs:http -r node1:http -m -w 1
> -a -t vs:http -r node2:http -m -w 1
> -a -t vs:http -r node3:http -m -w 1
>
> So persistence is disabled. The problem is of course the web servers are getting very uneven load, one is doing all the work while the others are idling around...
>
>
If you are using keepalived, your lvs.conf is irrelevant, and it's the
keepalived that you want to check.

>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

siim at p6drad-teel

Apr 29, 2009, 9:21 AM

Post #4 of 6 (1034 views)
Permalink

Re: [lvs-users] Unbalanced Real Servers [In reply to]

Götz Rieger wrote:
> So persistence is disabled. The problem is of course the web servers
> are getting very uneven load, one is doing all the work while the
> others are idling around...

Do all of the web servers have the same conf and hardware?

Inactive connections seem to be about even on all machines which
indicates that all real servers get the same amount of connections but
some of the real servers are keeping the connections open for a longer
time, for whatever reasons - some are refusing the connection, serving
the connection faster, have connection keepalive as suggested by Graeme
or something completely different.

Siim

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

goetz.rieger at gmx

Apr 29, 2009, 9:26 AM

Post #5 of 6 (1032 views)
Permalink

Re: [lvs-users] Unbalanced Real Servers [In reply to]

Hi Graeme,

Graeme Fowler wrote:
> On Wed, 2009-04-29 at 15:05 +0200, "Götz Rieger" wrote:
>> I'm doing a load test now using The Grinder with four load test clients. The problem I'm facing is an extremely uneven distribution of ActiveConn on the real servers:
>
> OK...
>
>> [root [at] lvst ~]# ipvsadm -L -n
>> IP Virtual Server version 1.2.1 (size=4096)
>> Prot LocalAddress:Port Scheduler Flags
>> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>> TCP 194.xxx.xxx.xxx:80 rr
>> -> 192.168.xxx.xxx:80 Masq 1 137 421
>> -> 192.168.xxx.xxx:80 Masq 1 203 509
>> -> 192.168.xxx.xxx:80 Masq 1 0 630
>
> A larger number of ActiveConns indicates a machine which is slower to
> respond (that is, these connections have not reached TIME_WAIT or
> FIN_WAIT), or a machine which has KeepAlive switched on in Apache.

Sure, but the servers and the OS are configured exactly the same. And
Apache _is_ the same on all nodes because it resides on the NFS-share...

KeepAlive is actually switched on, but again on all web servers.

So I really don't understand this behaviour right now. If the uneven
distribution would always follow the same pattern, yes. But the servers
getting high/low load are swapping around.

> Of more significance to you during testing would be:
>
> ipvsadm -L -n --stats
> ipvsadm -L -n --rate

I had a look at the --stats output as well. It looks like the throughput
is nearly the same for the real servers.

What concerns me is the web server/s with the high ActiveConns is/are
actually heavily loaded and the other/s are idle... and this makes the
whole thing a problem.

Might be some intricacy of TCP/IP I don't know about or the use of
NFS... but I don't get it.

> BTW I presume you meant "lc" scheduler rather than "ll"?

Err, yes. :-)

Those load test runs where actually ok with the ActiveConns staying the
same all the time... so should I go with lc?

Thanks for replying.

Goetz

>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

goetz.rieger at gmx

Apr 29, 2009, 9:29 AM

Post #6 of 6 (1032 views)
Permalink

Re: [lvs-users] Unbalanced Real Servers [In reply to]

Hi,

Thomas Pedoussaut wrote:
> Götz Rieger wrote:
>> The configuration is as easy as it gets:
>>
>> [root [at] lvst ~]# cat lvs.conf
>> -A -t vs:http -s rr
>> -a -t vs:http -r node1:http -m -w 1
>> -a -t vs:http -r node2:http -m -w 1
>> -a -t vs:http -r node3:http -m -w 1
>>
>> So persistence is disabled. The problem is of course the web servers are getting very uneven load, one is doing all the work while the others are idling around...
>>
>>
> If you are using keepalived, your lvs.conf is irrelevant, and it's the
> keepalived that you want to check.

This was just to show the configuration in an easy way and the name was
just by chance used for the output of ipvsadm-save.

The actual configuration is of course done in keepalived.conf.

Cheers,
Goetz

>
>>
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads