Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS logging

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


ABhat at trustwave

Apr 2, 2009, 11:06 AM

Post #1 of 5 (813 views)
Permalink
[lvs-users] LVS logging

Hi There.

Just wanted to say, I really love LVS-DR. It's fantastic!0 A bit confusing to wrap your head around but I find it much simpler than LVS-NAT.

There is some concern that our engineers have expressed about losing connections to the real servers.

Currently, we have a pair of LVS servers running 21 services (mostly http and https to apache & tomcat). These servers get connections from agents we've deployed all over the country (about 10000) every five minutes.

The servers are Centos 5.2 running piranha.

We're wondering if there are connections that make it to LVS that somehow don't get redirected to the real servers.

Is there a way to determine if this is happening and also to gather any metric information on these occurrences.

Thanks

Anoop Bhat


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Apr 2, 2009, 11:25 AM

Post #2 of 5 (757 views)
Permalink
Re: [lvs-users] LVS logging [In reply to]

On Thu, 2 Apr 2009, Anoop Bhat wrote:

> We're wondering if there are connections that make it to LVS that somehow don't get redirected to the real servers.

For LVS-NAT and LVS-Dr, LVS generates icmp packets so that
the connection obeys tcpip semantics. If you're loosing
packets you could use the same methods as you would for a
standalone server

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ABhat at trustwave

Apr 2, 2009, 11:40 AM

Post #3 of 5 (759 views)
Permalink
Re: [lvs-users] LVS logging [In reply to]

For a standalone server I would normally initiate the connection and verify using tcpdump.

But for the number of connections we get, it would be impossible to verify every connection wouldn't it?

Anoop Bhat
Systems Administrator
Trustwave
70 W. Madison
Chicago, IL, 60602
O: 312.873.7446
C: 312.925.3271



________________________________
From: Joseph Mack NA3T <jmack [at] wm7d>
Reply-To: "LinuxVirtualServer.org users mailing list." <lvs-users [at] linuxvirtualserver>
Date: Thu, 2 Apr 2009 13:25:11 -0500
To: "LinuxVirtualServer.org users mailing list." <lvs-users [at] linuxvirtualserver>
Subject: Re: [lvs-users] LVS logging

On Thu, 2 Apr 2009, Anoop Bhat wrote:

> We're wondering if there are connections that make it to LVS that somehow don't get redirected to the real servers.

For LVS-NAT and LVS-Dr, LVS generates icmp packets so that
the connection obeys tcpip semantics. If you're loosing
packets you could use the same methods as you would for a
standalone server

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users




_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Apr 2, 2009, 12:01 PM

Post #4 of 5 (761 views)
Permalink
Re: [lvs-users] LVS logging [In reply to]

On Thu, 2 Apr 2009, Anoop Bhat wrote:

> For a standalone server I would normally initiate the connection and verify using tcpdump.
>
> But for the number of connections we get, it would be impossible to verify every connection wouldn't it?

do you mean you want at test for LVS when you don't have a
test for a standalone server?

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Apr 2, 2009, 12:25 PM

Post #5 of 5 (771 views)
Permalink
Re: [lvs-users] LVS logging [In reply to]

On Thu, 2009-04-02 at 13:06 -0500, Anoop Bhat wrote:
> The servers are Centos 5.2 running piranha.

OK... so this question may be better off being asked of the Piranha
mailing list, which has been mentioned on numerous occasions on this
list. Relatively few subscribers to this list use Piranha so far as we
can tell.

> We're wondering if there are connections that make it to LVS that somehow don't get redirected to the real servers.

Possibly, but it depends on what Piranha (and the constituent parts
thereof) are doing.

> Is there a way to determine if this is happening and also to gather any metric information on these occurrences.

There might be, but with 10000 clients you *really* don't want to have
to switch on LVS debugging, believe me! It's also very likely that the
CentOS kernel is not compiled to support LVS debugging.

You'll probably end up looking at the bits of Piranha that do health
checking. As a non-user, I have no idea what they are - mon, perhaps,
along with heartbeat for failover? I'd guess you can switch something on
there to produce debug logs.

At a more basic level, run tcpdump on a few of the clients (or the
equivalent) and also on the directors and realservers, filtering for the
client source addresses. If you're able to keep all the clocks in sync
and have enough space to store the pcap files, you should be able to
trap enough data to correlate where the traffic is disappearing.

Also worth considering is whether or not you have the netfilter
(iptables) conntrack modules loaded somewhere with a table size set too
small... that might cause connections to be flushed from an ESTABLISHED
state before they really are finished, thus causing the connection to be
dropped.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.