Mailing List Archive: Linux Virtual Server: Users

[lvs-users] Ghost connections

Index | Next | Previous | View Threaded

fxmulder at gmail

Sep 28, 2009, 1:40 PM

Post #1 of 8 (1356 views)
Permalink

[lvs-users] Ghost connections

I have heartbeat/ldirectord setup to balance web traffic between 2 web
servers. I noticed that ipvsadm -Lnc is listing established
connections that the destination server is saying do not exist when I
grep for the source IP in netstat -an. Any idea why these connections
aren't getting cleared out?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Sep 28, 2009, 3:39 PM

Post #2 of 8 (1285 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

On Mon, 2009-09-28 at 14:40 -0600, James Devine wrote:
> I have heartbeat/ldirectord setup to balance web traffic between 2 web
> servers. I noticed that ipvsadm -Lnc is listing established
> connections that the destination server is saying do not exist when I
> grep for the source IP in netstat -an. Any idea why these connections
> aren't getting cleared out?

If you're running netstat on the director you won't see them as the
sessions are not established at that level on the director. The session
table seen by ipvsadm is at ip_vs level, not IP - it's application
agnostic.

If you're running netstat on the realserver and it's already gone
through the various WAIT states and timed out, then you're using -DR and
the director isn't seeing the full conversation so doesn't know what's
going on. The sessions seen under "ipvsadm -Lnc" will disappear after (I
think) two minutes, unless you're using persistence at which point it
depends on your configuration.

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Sep 28, 2009, 3:42 PM

Post #3 of 8 (1304 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

On Mon, Sep 28, 2009 at 02:40:26PM -0600, James Devine wrote:
> I have heartbeat/ldirectord setup to balance web traffic between 2 web
> servers. I noticed that ipvsadm -Lnc is listing established
> connections that the destination server is saying do not exist when I
> grep for the source IP in netstat -an. Any idea why these connections
> aren't getting cleared out?

Hi James,

what state are these connections in?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

fxmulder at gmail

Sep 29, 2009, 6:48 AM

Post #4 of 8 (1276 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

The connections are all in an established state on the load balancer,
I am doing netstat -an on the actual destination real server, which
does not show any connections from the real source ip. I am using
persistent connections, which do show an expiration on the
connections. Is the persistent setting why they are not clearing?
Some IPs show >300 established connections when the realserver shows
none.

On Mon, Sep 28, 2009 at 4:42 PM, Simon Horman <horms [at] verge> wrote:
> On Mon, Sep 28, 2009 at 02:40:26PM -0600, James Devine wrote:
>> I have heartbeat/ldirectord setup to balance web traffic between 2 web
>> servers. �I noticed that ipvsadm -Lnc is listing established
>> connections that the destination server is saying do not exist when I
>> grep for the source IP in netstat -an. �Any idea why these connections
>> aren't getting cleared out?
>
> Hi James,
>
> what state are these connections in?
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Sep 29, 2009, 7:44 AM

Post #5 of 8 (1276 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

On Tue, Sep 29, 2009 at 07:48:55AM -0600, James Devine wrote:
> The connections are all in an established state on the load balancer,
> I am doing netstat -an on the actual destination real server, which
> does not show any connections from the real source ip. I am using
> persistent connections, which do show an expiration on the
> connections. Is the persistent setting why they are not clearing?

No, I don't think so.

Activating persistence means that persistence templates will be created for
each client that connects. These can be identified as entries in the
connection table with 0 as the source port. However these entries should
be in the NONE state, so they shouldn't causing the behaviour that you are
observing.

> Some IPs show >300 established connections when the realserver shows
> none.

That does sound somewhat curious. Is that >300 established connections
from a single end-user?

I would tend to suspect that actually are being terminated somewhere
other than the real-server you are checking.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

fxmulder at gmail

Sep 29, 2009, 8:21 AM

Post #6 of 8 (1271 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

The IP that had >300 connections looked like a large facility so it
may have been multiple users. Right now I am seeing a max of 19
established connections from a single IP, those connections aren't
present on the end realserver though. I do see one NONE entry for
that IP.

What do you mean terminating somewhere other than the realserver?
Terminating from the source?

On Tue, Sep 29, 2009 at 8:44 AM, Simon Horman <horms [at] verge> wrote:
> On Tue, Sep 29, 2009 at 07:48:55AM -0600, James Devine wrote:
>> The connections are all in an established state on the load balancer,
>> I am doing netstat -an on the actual destination real server, which
>> does not show any connections from the real source ip. �I am using
>> persistent connections, which do show an expiration on the
>> connections. �Is the persistent setting why they are not clearing?
>
> No, I don't think so.
>
> Activating persistence means that persistence templates will be created for
> each client that connects. These can be identified as entries in the
> connection table with 0 as the source port. �However these entries should
> be in the NONE state, so they shouldn't causing the behaviour that you are
> observing.
>
>> Some IPs show >300 established connections when the realserver shows
>> none.
>
> That does sound somewhat curious. Is that >300 established connections
> from a single end-user?
>
> I would tend to suspect that actually are being terminated somewhere
> other than the real-server you are checking.
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Sep 29, 2009, 5:08 PM

Post #7 of 8 (1273 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

On Tue, Sep 29, 2009 at 09:21:36AM -0600, James Devine wrote:
> The IP that had >300 connections looked like a large facility so it
> may have been multiple users. Right now I am seeing a max of 19
> established connections from a single IP, those connections aren't
> present on the end realserver though. I do see one NONE entry for
> that IP.
>
> What do you mean terminating somewhere other than the realserver?
> Terminating from the source?

What I was thinking is that perhaps those connections are being
received by a daemon running on a host other than the real-server
you are checking. Perhaps because some other host has its IP address.
Perhaps the linux-director? Its just a stab in the dark.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

fxmulder at gmail

Sep 30, 2009, 8:04 AM

Post #8 of 8 (1254 views)
Permalink

Re: [lvs-users] Ghost connections [In reply to]

Well the way it is setup is the virtual IP is tied to localhost on
each of the real servers. The real ip on the real servers only exists
on the real servers. I do catch some connections that the load
balancer says are established which are in FIN_WAIT2 on the real
server.

New connections seem to come in with a 15 minute expire, some
connections seem to go into FIN_WAIT after a few minutes of it
clearing off of the real server at which point their expiration time
changes to 2 minutes, some seem to wait the full 15 minutes in an
established state.

On Tue, Sep 29, 2009 at 6:08 PM, Simon Horman <horms [at] verge> wrote:
> On Tue, Sep 29, 2009 at 09:21:36AM -0600, James Devine wrote:
>> The IP that had >300 connections looked like a large facility so it
>> may have been multiple users. �Right now I am seeing a max of 19
>> established connections from a single IP, those connections aren't
>> present on the end realserver though. �I do see one NONE entry for
>> that IP.
>>
>> What do you mean terminating somewhere other than the realserver?
>> Terminating from the source?
>
> What I was thinking is that perhaps those connections are being
> received by a daemon running on a host other than the real-server
> you are checking. Perhaps because some other host has its IP address.
> Perhaps the linux-director? Its just a stab in the dark.
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads