jfjm2002 at yahoo
Mar 3, 2009, 2:17 AM
Post #5 of 7
(1079 views)
Permalink
|
Let me rephrase my first question below. What I would like to know is how well can LVS code scale with multiple CPU. I do not know that balancing out IRQ with the NIC works pretty well with Linux 2.6.18 kernel. Certain Broadcom NICs seem to have problem with MSI turned on, but Intel NICs works fine. So, *assume* the kernel can linearly scale up with SMP with NICs' interrupts, can the LVS code do the same?
Jason, handling of NIC interrupts does balance over multiple CPU when you either manually set the SMP affinity or use irqbalance.
Graeme, it's a good thing the CPU is being used, but I expect our traffic to increase dramatically, so there is not much room left for growth. That's why we are considering something much "beefier" to handle the load. The load (1/5/15) average on the machine is almost nothing when CPU usage is at 160%. It is mostly taken up by Soft IRQ.
----- Original Message ----
From: New User <jfjm2002 [at] yahoo>
To: lvs-users [at] linuxvirtualserver
Sent: Tuesday, March 3, 2009 1:45:12 PM
Subject: LVS performance on SMP
Hi,
We are currently running dedicated LVS-NAT (with the exception of iptables):
Intel Xeon 3060 @ 2.4GHz (2 cores)
2G RAM
2 Gb/s NIC (one external, one internal)
LVS-NAT
Linux 2.6.18 kernel
It appears we are running out of CPU (usage reached 160, with 40 idle, 2 cores) when we reach:
~10K CPS
~100K InPPS
~120K OutPPS
~1 million+ active and number of objects in the ip_vs_conn entry in /proc/slabinfo
We are considering upgrading the setup to the following:
2 x Intel Xeon 5430 (2 CPUs, 4 cores each)
8G RAM
8 Gb/s NIC (4x1Gb in bonding for external, 4x1Gb in bonding for internal)
LVS-NAT
Linux 2.6.18 (possible newer if it offers performance increase)
The question is how well can the LVS code take advantages of the multiple CPU. There are some conflicting answers to this question on the FAQ and mailing list (in gernal, it helps with the system as a whole). In our actual usage experience, it appears SMP helps, since it did reach 160 CPU usage. However, I do not know if this is real or just an illusion. I do not know if LVS is actually doing real work that caused the usage to be above 100 or is it because of SMP that causing locks that resulted that. Also, in our limited testing of the new setup, we find the SMP helps because with 8 NICs, just to process the interrupts can overwhelm a single CPU (core).
Can anyone share their knowledge or experience for the following:
1. Can LVS increase linearly in capacity when we go from the setup we currently have (2 cores with 2 NICs) to what we are considering (8 cores, with 8 NICs)
2. Has anyone else used similiar setup that can share their experience? There was a posting to the mailing list in Dec. 2007 that indicated the performance limit is ~450K (don't know if it's the sum of InPPS and OutPPS) pps. There was no conclusive answer as to this was the limit of LVS or not
3. Has anyone have a setup that can process 1+ million pps (LVS-NAT) on a single LVS machine (we already use multipel LVS with keepalived in a live-live configuration)?
Thanks
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
