Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] Connections are "clumped" together on real_server failure

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


jay.faulkner at mailtrust

Jun 25, 2009, 7:23 AM

Post #1 of 2 (590 views)
Permalink
[lvs-users] Connections are "clumped" together on real_server failure

Hey guys,

We've been noticing some really strange behavior with keepalived/LVS when a real_server fails. The behavior we see is that when a r_s fails, all the persistent connections on that server fall over to a single server.

For example:

Server 1 - 25000 active conn. - 1500 inactive conn.
Server 2 - 35000 active conn. - 1000 inactive conn.
Server 3 - 20000 active conn. - 500 inactive conn.

Server 2 fails. Instead of seeing all the traffic split between the remaining real_servers, we see it all clump over to the same server

Server 1 - 25000 active conn. - 1500 inactive conn.
Server 2 - failed
Server 3 - 55000 active conn. - 1500 inactive conn.

Now, server 3 falls over and dies due to having too many connections on it and becoming overloaded.

We've seen this behavior consistently in all of our keepalived installations (20 servers or so? Median of about 20 virtual_servers each), and I was curious if this is configurable behavior, or even expected.

Thanks!

Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner [at] rackspace<mailto:jason.faulkner [at] rackspace>
o: (540) 443-2101 (ex. 505-2101)

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


malcolm at loadbalancer

Jun 30, 2009, 2:37 AM

Post #2 of 2 (541 views)
Permalink
Re: [lvs-users] Connections are "clumped" together on real_server failure [In reply to]

Jay,

Try switching from WLC to WRR? (I'm guessing that you are using WLC).



2009/6/25 Jay Faulkner <jay.faulkner [at] mailtrust>:
> Hey guys,
>
> We've been noticing some really strange behavior with keepalived/LVS when a real_server fails. The behavior we see is that when a r_s fails, all the persistent connections on that server fall over to a single server.
>
> For example:
>
> Server 1 - 25000 active conn. - 1500 inactive conn.
> Server 2 - 35000 active conn. - 1000 inactive conn.
> Server 3 - 20000 active conn. - 500 inactive conn.
>
> Server 2 fails. Instead of seeing all the traffic split between the remaining real_servers, we see it all clump over to the same server
>
> Server 1 - 25000 active conn. - 1500 inactive conn.
> Server 2 - failed
> Server 3 - 55000 active conn. - 1500 inactive conn.
>
> Now, server 3 falls over and dies due to having too many connections on it and becoming overloaded.
>
> We've seen this behavior consistently in all of our keepalived installations (20 servers or so? Median of about 20 virtual_servers each), and I was curious if this is configurable behavior, or even expected.
>
> Thanks!
>
> Jason Faulkner
> Linux Engineer, Rackspace Email & Apps
> jason.faulkner [at] rackspace<mailto:jason.faulkner [at] rackspace>
> o: (540) 443-2101 (ex. 505-2101)
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.