Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS-DR Director not passing some packets

Index | Next | Previous | View Threaded

dave at jsdi

Mar 10, 2008, 3:39 AM

Post #1 of 4 (251 views)
Permalink

[lvs-users] LVS-DR Director not passing some packets

Just looking for some advice around problem solving my LVS cluster
performance.

I have a cluster that currently has two LVS Directors hosting 4 VIPS. I use
keepalived 1.1.13 on the directors, running on CentOS 4.4. Real servers are
all CentOS 4.4 too.

Basically balancing web traffic with Directed Routing and Round Robin load
balance.

I have hit an issue where my directors appear to be dropping packets. My
connection rate is low when this occurs, typically about 40+ InPPS which is
very low :-(

A packet trace appears to indicate that packets are hitting the director but
not coming out again.

I have iptables running on the directors and ip_contrack is loaded along
with a few other iptable related kernel modules. My question is do you thing
that iptables could be causing problems with my LVS and if so have you any
suggestions where and how I could prove it ?

Thanks

Dave

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Mar 10, 2008, 6:30 AM

Post #2 of 4 (242 views)
Permalink

Re: [lvs-users] LVS-DR Director not passing some packets [In reply to]

On Mon, 10 Mar 2008, Dave Whitehouse wrote:

> I have iptables running on the directors and ip_contrack is loaded along
> with a few other iptable related kernel modules.

ip_conntrack is incompatible with getting good performance
(I seem to remember that throughtput is about half).

> My question is do you thing
> that iptables could be causing problems with my LVS and if so have you any
> suggestions where and how I could prove it ?

remove the iptables rules

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Mar 11, 2008, 2:59 AM

Post #3 of 4 (242 views)
Permalink

Re: [lvs-users] LVS-DR Director not passing some packets [In reply to]

Hi

Just to add to Joe's comments...

On Mon, 2008-03-10 at 10:39 +0000, Dave Whitehouse wrote:
> Just looking for some advice around problem solving my LVS cluster
> performance.
>
> I have a cluster that currently has two LVS Directors hosting 4 VIPS. I use
> keepalived 1.1.13 on the directors, running on CentOS 4.4. Real servers are
> all CentOS 4.4 too.

CentOS 4 is a bit behind the bleeding edge but that does give it the
benefit of being somewhat more stable than, for example, Fedora :)

> I have hit an issue where my directors appear to be dropping packets. My
> connection rate is low when this occurs, typically about 40+ InPPS which is
> very low :-(

That is fairly low.

> A packet trace appears to indicate that packets are hitting the director but
> not coming out again.

As in, not being forwarded to the realservers?

> I have iptables running on the directors and ip_contrack is loaded along
> with a few other iptable related kernel modules. My question is do you thing
> that iptables could be causing problems with my LVS and if so have you any
> suggestions where and how I could prove it ?

Well, the easiest thing to do is to remove all the iptables rules and
unload the modules, then test. If the same problem occurs, you've found
that it isn't iptables and conversely if it doesn't, then it is.

You can then build your ruleset back up bit by bit and see which part
causes the problem.

For us to help you further we probably need to see your keepalived
config, the output from ipvsadm in normal operation and the output from
ipvsadm when there's a problem.

Also, have you "cloaked" the realservers to prevent them repsonding to
ARP requests for the VIP?

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

dave at jsdi

Mar 21, 2008, 10:58 AM

Post #4 of 4 (229 views)
Permalink

Re: [lvs-users] LVS-DR Director not passing some packets [In reply to]

We have found some other issues that could have been causing the real
servers to fail the health check intermittently. This obviously wouldn't
help matter.

However the packet trace did appear to point to the director not forwarding
to the real server.

I have setup a similar config in a test lab and have got 33000 InPPS with
no problems so I'm thinking it is not LVS related at the moment.

Thanks for the input thou' guys and I'll keep you posted.

-----Original Message-----
From: lvs-users-bounces[at]linuxvirtualserver.org
[mailto:lvs-users-bounces[at]linuxvirtualserver.org] On Behalf Of Graeme Fowler
Sent: 11 March 2008 10:00
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] LVS-DR Director not passing some packets

Hi

Just to add to Joe's comments...

On Mon, 2008-03-10 at 10:39 +0000, Dave Whitehouse wrote:
> Just looking for some advice around problem solving my LVS cluster
> performance.
>
> I have a cluster that currently has two LVS Directors hosting 4 VIPS. I
use
> keepalived 1.1.13 on the directors, running on CentOS 4.4. Real servers
are
> all CentOS 4.4 too.

CentOS 4 is a bit behind the bleeding edge but that does give it the
benefit of being somewhat more stable than, for example, Fedora :)

> I have hit an issue where my directors appear to be dropping packets. My
> connection rate is low when this occurs, typically about 40+ InPPS which
is
> very low :-(

That is fairly low.

> A packet trace appears to indicate that packets are hitting the director
but
> not coming out again.

As in, not being forwarded to the realservers?

> I have iptables running on the directors and ip_contrack is loaded along
> with a few other iptable related kernel modules. My question is do you
thing
> that iptables could be causing problems with my LVS and if so have you any
> suggestions where and how I could prove it ?

Well, the easiest thing to do is to remove all the iptables rules and
unload the modules, then test. If the same problem occurs, you've found
that it isn't iptables and conversely if it doesn't, then it is.

You can then build your ruleset back up bit by bit and see which part
causes the problem.

For us to help you further we probably need to see your keepalived
config, the output from ipvsadm in normal operation and the output from
ipvsadm when there's a problem.

Also, have you "cloaked" the realservers to prevent them repsonding to
ARP requests for the VIP?

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com