Mailing List Archive: Linux Virtual Server: Users

[lvs-users] TCP connection dropping ~7% of the time

Index | Next | Previous | View Threaded

jay.faulkner at mailtrust

Jul 21, 2009, 8:49 AM

Post #1 of 10 (1567 views)
Permalink

[lvs-users] TCP connection dropping ~7% of the time

Hey guys,

I have an LVS-NAT configuration of two http servers configured with persistence. These servers are being reverse-proxied to, and so all the connections are coming from a single IP. Effectively, we'll have one server with ~3000 inactive conn, and the other with 0. While this configuration is a little bit silly, admittedly, it was working fine for a small amount of time.

The problem we're experiencing now is that somewhere between 3% and 7% of all connections are dropping - same behavior you'd see with an iptables DROP rule or a missing return route. We aren't seeing the issue when we transition from a LVS to a direct DNAT to a VIP.

Any ideas?

Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner [at] rackspace<mailto:jason.faulkner [at] rackspace>
o: (540) 443-2101 (ex. 505-2101)

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Jul 21, 2009, 12:56 PM

Post #2 of 10 (1481 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

On Tue, 21 Jul 2009, Jay Faulkner wrote:

> Hey guys,
>
> The problem we're experiencing now is that somewhere
> between 3% and 7% of all connections are dropping -

you don't say what "dropping" is. Is it entries disappearing
from the ipvsadm table? clients getting RST?

> same behavior you'd see with an iptables DROP rule or a
> missing return route. We aren't seeing the issue when we
> transition from a LVS to a direct DNAT to a VIP.

this is a wierd one. Looks a bit like arp hopping, so
let's assume it's a routing problem. Let's look for wierd
reasons.

o make sure there are no iptables rules on any of the
machines.

o test with local client (not through proxy) connecting to
VIP

o make sure there is no physical route from the realserver
to the client except throught the DIP - ie no routes that an
imcp redirect that change. check your routes with

ip route show

(not `route -n`)

o test with a simple client. telnet is best (to port 23)
then next best is telnet to port 80 and do a GET

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jay.faulkner at mailtrust

Jul 21, 2009, 2:17 PM

Post #3 of 10 (1484 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

Comments inline.

Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner [at] rackspace
o: (540) 443-2101 (ex. 505-2101)

> > The problem we're experiencing now is that somewhere
> > between 3% and 7% of all connections are dropping -
>
> you don't say what "dropping" is. Is it entries disappearing
> from the ipvsadm table? clients getting RST?
>

I'm not sure, to be honest. This is the "needle" in the haystack -- out of maybe 60k connections, 1 will fail, and I've yet to find it on a packet capture.

> > same behavior you'd see with an iptables DROP rule or a
> > missing return route. We aren't seeing the issue when we
> > transition from a LVS to a direct DNAT to a VIP.
>
> this is a wierd one. Looks a bit like arp hopping, so
> let's assume it's a routing problem. Let's look for wierd
> reasons.
>
> o make sure there are no iptables rules on any of the
> machines.
>

Iptables rules are on the machine, but didn't change at all between it working and it not working.

> o test with local client (not through proxy) connecting to
> VIP
>
> o make sure there is no physical route from the realserver
> to the client except throught the DIP - ie no routes that an
> imcp redirect that change. check your routes with
>
> ip route show
>
> (not `route -n`)
>
> o test with a simple client. telnet is best (to port 23)
> then next best is telnet to port 80 and do a GET
>

We've never reproduced the problem hitting the VIP directly from anywhere but the server the proxy lives on. From that server, we see basic connection failures, similar to an iptables drop, regardless of client (we wrote an automated python testing tool).

The routes are clean, and the traceroute is the same both ways (so there isn't a circular routing problem).

This looks so much like it has to be an issue with the network on the side of the proxy server; but as soon as we moved it to a DNAT the problem resolved itself. My conjecture has been that perhaps ipvs has some sort of limit on the number of active/inactive connections sourcing from the same IP address?

Thanks,
Jay

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

malcolm at loadbalancer

Jul 22, 2009, 3:27 AM

Post #4 of 10 (1476 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

"of maybe 60k connections"
Interesting, this is the kind of level that the conntrack table can
get filled up?
Are you sure you are not getting kernel level messages on either the
proxy, real server or load balancer?

2009/7/21 Jay Faulkner <jay.faulkner [at] mailtrust>
>
> Comments inline.
>
> Jason Faulkner
> Linux Engineer, Rackspace Email & Apps
> jason.faulkner [at] rackspace
> o: (540) 443-2101 (ex. 505-2101)

--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Jul 22, 2009, 4:39 AM

Post #5 of 10 (1474 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

On Wed, 2009-07-22 at 11:27 +0100, Malcolm Turnbull wrote:
> Interesting, this is the kind of level that the conntrack table can
> get filled up?

Ah, good spot, Malcolm - alternatively (and this is doubtful, but could
be true) it's also possible under a load of many (thousands of?)
connections/second that the IPVS connection table could be overflowing.

I'd lean ore towards the conntrack table, though.

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Jul 22, 2009, 5:33 AM

Post #6 of 10 (1465 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

On Wed, 22 Jul 2009, Graeme Fowler wrote:

> it's also possible under a load of many (thousands of?)
> connections/second that the IPVS connection table could be
> overflowing.

this won't happen till there are ($your_memory)/128byte
connections

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jay.faulkner at mailtrust

Jul 22, 2009, 8:29 AM

Post #7 of 10 (1472 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

Iptables conntrack is set to accept up to 1,000,000 connections, and the box has a ton of RAM.

In addition, this is one of our smallest V_S... we have some of these that pull over 400,000 active connections at a time, no problem (on that same LB). That's part of the reason this problem is so incredibly perplexing.

Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner [at] rackspace
o: (540) 443-2101 (ex. 505-2101)

> -----Original Message-----
> From: lvs-users-bounces [at] linuxvirtualserver [mailto:lvs-users-
> bounces [at] linuxvirtualserver] On Behalf Of Joseph Mack NA3T
> Sent: Wednesday, July 22, 2009 8:33 AM
> To: LinuxVirtualServer.org users mailing list.
> Subject: Re: [lvs-users] TCP connection dropping ~7% of the time
>
> On Wed, 22 Jul 2009, Graeme Fowler wrote:
>
> > it's also possible under a load of many (thousands of?)
> > connections/second that the IPVS connection table could be
> > overflowing.
>
> this won't happen till there are ($your_memory)/128byte
> connections
>
> Joe
>
> --
> Joseph Mack NA3T EME(B,D), FM05lw North Carolina
> jmack (at) wm7d (dot) net - azimuthal equidistant map
> generator at http://www.wm7d.net/azproj.shtml
> Homepage http://www.austintek.com/ It's GNU/Linux!
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

heder at google

Jul 23, 2009, 8:32 AM

Post #8 of 10 (1477 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

On Wed, Jul 22, 2009 at 14:33, Joseph Mack NA3T<jmack [at] wm7d> wrote:
> On Wed, 22 Jul 2009, Graeme Fowler wrote:
>
>> it's also possible under a load of many (thousands of?)
>> connections/second that the IPVS connection table could be
>> overflowing.
>
> this won't happen till there are ($your_memory)/128byte
> connections

128 bytes are no longer true, at least since IPv6 support for IPVS was
merged, but other things have changed as well. In my config it's
264 bytes, but millage may vary.

diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index b021464..f5538f4 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -1476,6 +1476,8 @@ static int __init ip_vs_init(void)
}

IP_VS_INFO("ipvs loaded.\n");
+ printk(KERN_INFO "IPVS: sizeof(struct ip_vs_conn)=%Zd\n",
+ sizeof(struct ip_vs_conn));
return ret;

cleanup_conn:

Cheers,
-Hannes

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

malcolm at loadbalancer

Jul 23, 2009, 11:23 AM

Post #9 of 10 (1455 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

Hannes,

I forgot about that change, It is permanent then?
I heard some one was going to reduce it for IPV4.

That's a big chunk, doubling the amount of memory required per connection...
Still memory is cheap I guess.

2009/7/23 Hannes Eder <heder [at] google>:
> On Wed, Jul 22, 2009 at 14:33, Joseph Mack NA3T<jmack [at] wm7d> wrote:
>> On Wed, 22 Jul 2009, Graeme Fowler wrote:
>>
>>> it's also possible under a load of many (thousands of?)
>>> connections/second that the IPVS connection table could be
>>> overflowing.
>>
>> this won't happen till there are ($your_memory)/128byte
>> connections
>
> 128 bytes are no longer true, at least since IPv6 support for IPVS was
> merged, but other things have changed as well. �In my config it's
> 264 bytes, but millage may vary.
>
> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
> index b021464..f5538f4 100644
> --- a/net/netfilter/ipvs/ip_vs_core.c
> +++ b/net/netfilter/ipvs/ip_vs_core.c
> @@ -1476,6 +1476,8 @@ static int __init ip_vs_init(void)
> � � � �}
>
> � � � �IP_VS_INFO("ipvs loaded.\n");
> + � � � printk(KERN_INFO "IPVS: sizeof(struct ip_vs_conn)=%Zd\n",
> + � � � � � � �sizeof(struct ip_vs_conn));
> � � � �return ret;
>
> � cleanup_conn:
>
> Cheers,
> -Hannes
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

heder at google

Jul 27, 2009, 2:05 AM

Post #10 of 10 (1417 views)
Permalink

Re: [lvs-users] TCP connection dropping ~7% of the time [In reply to]

On Thu, Jul 23, 2009 at 20:23, Malcolm Turnbull<malcolm [at] loadbalancer> wrote:
> I forgot about that change, It is permanent then?

IPv6 for IPVS is mainline, see:

commit e7ade46a53055c19a01c8becbe7807f9075d6fee
Author: Julius Volz <juliusv [at] google>
Date: Tue Sep 2 15:55:33 2008 +0200

this contributes 2 (+ u16 af) + 3 * 12 (union nf_inet_addr instead of
__be32) = 50 bytes / connection, other changes contribute apparently
more.

Cheers,
Hannes

> I heard some one was going to reduce it for IPV4.
>
> That's a big chunk, doubling the amount of memory required per connection...
> Still memory is cheap I guess.
>
>
>
>
>
>
> 2009/7/23 Hannes Eder <heder [at] google>:
>> On Wed, Jul 22, 2009 at 14:33, Joseph Mack NA3T<jmack [at] wm7d> wrote:
>>> On Wed, 22 Jul 2009, Graeme Fowler wrote:
>>>
>>>> it's also possible under a load of many (thousands of?)
>>>> connections/second that the IPVS connection table could be
>>>> overflowing.
>>>
>>> this won't happen till there are ($your_memory)/128byte
>>> connections
>>
>> 128 bytes are no longer true, at least since IPv6 support for IPVS was
>> merged, but other things have changed as well. �In my config it's
>> 264 bytes, but millage may vary.
>>
>> diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
>> index b021464..f5538f4 100644
>> --- a/net/netfilter/ipvs/ip_vs_core.c
>> +++ b/net/netfilter/ipvs/ip_vs_core.c
>> @@ -1476,6 +1476,8 @@ static int __init ip_vs_init(void)
>> � � � �}
>>
>> � � � �IP_VS_INFO("ipvs loaded.\n");
>> + � � � printk(KERN_INFO "IPVS: sizeof(struct ip_vs_conn)=%Zd\n",
>> + � � � � � � �sizeof(struct ip_vs_conn));
>> � � � �return ret;
>>
>> � cleanup_conn:
>>
>> Cheers,
>> -Hannes
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
>> Send requests to lvs-users-request [at] LinuxVirtualServer
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)870 443 8779
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads