Mailing List Archive: Linux Virtual Server: Users

LVS DR increased "system CPU" and Excessive Network Traffic

Index | Next | Previous | View Threaded

iain at g7iii

Apr 4, 2007, 12:35 PM

Post #1 of 6 (608 views)
Permalink

LVS DR increased "system CPU" and Excessive Network Traffic

Hi Folks,

I've just installed a couple of machines with LVS on, using the
DR method. I also used the localnode feature, and started the
sync daemon as master on one, and backup on the other.

The nodes are functioning both as Director, and Real Servers,
and now, both machines are:

a) showing "System CPU" at around 20%, which vanishes if I kill
LVS.

b) Sending out an awful lot of LAN traffic (120,000 packets an
hour!), which I'm guessing is multicast [which is to be expected],
but surely it doesnt need *that* many packets! Again, it dissapears
when I shutdown LVS.

Curiously, even if I start LVS w/o running the master and backup
daemons, then I also still both effects. This is on Ubuntu Feisty,
kernel 2.6.20, ipvsadm 1.24

Anyone have any ideas if this is a) Normal, b) What I might have
missed, c) How to fix it ?

I could move the LAN traffic off to another interface, and a crossover
cable, but that really only moves the problem, and doesnt sort the
CPU system issue.

Best Regards

Iain

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

stepedino at bhsoftware

Apr 4, 2007, 1:02 PM

Post #2 of 6 (594 views)
Permalink

Re: LVS DR increased "system CPU" and Excessive Network Traffic [In reply to]

On Wed, 2007-04-04 at 20:35 +0100, Iain Young wrote:
> a) showing "System CPU" at around 20%, which vanishes if I kill
> LVS.

This shouldn't be. My systems sit at idle, basically, waiting to do
something. The heartbeating should be of minimal impact.
What method are you using for director/real server failover? keepalived,
heartbeat/ldirectord or something else? It would help if you posted
configs.

> b) Sending out an awful lot of LAN traffic (120,000 packets an
> hour!), which I'm guessing is multicast [which is to be expected],
> but surely it doesnt need *that* many packets! Again, it dissapears
> when I shutdown LVS.

Depending on your setup, you should have, lets say, 1 packet per machine
per second. Perhaps 2 if you use two different heartbeat methods. So,
say, about 3600/server * 2, 7200, * 2 for multiple heartbeat methods,
14400... Now lets assume that because I don't pay much attention to the
packets flying back and forth between the machines I'm off by alot... I
doubt I'm off by an order of magnitude... am I?

> Curiously, even if I start LVS w/o running the master and backup
> daemons, then I also still both effects. This is on Ubuntu Feisty,
> kernel 2.6.20, ipvsadm 1.24
>
>
> Anyone have any ideas if this is a) Normal, b) What I might have
> missed, c) How to fix it ?

A) Not likely
B & C) Not really sure without seeing configs or knowing what method
you're using.

> I could move the LAN traffic off to another interface, and a crossover
> cable, but that really only moves the problem, and doesnt sort the
> CPU system issue.

Exactly. There's something else going on here. Masking the problem won't
do much good. Do the directors and real servers (I realize they're the
same box, I more mean the services) act as expected? Do they fail over
from director to director? Do real servers get removed as expected when
shutdown?

--
Sal Tepedino <stepedino [at] bhsoftware>
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

ipvsuser at itsbeen

Apr 4, 2007, 6:10 PM

Post #3 of 6 (577 views)
Permalink

Re: LVS DR increased "system CPU" and Excessive Network Traffic [In reply to]

Iain Young wrote:
> Hi Folks,
>
> I've just installed a couple of machines with LVS on, using the
> DR method. I also used the localnode feature, and started the
> sync daemon as master on one, and backup on the other.
>
> The nodes are functioning both as Director, and Real Servers,
> and now, both machines are:
>
> a) showing "System CPU" at around 20%, which vanishes if I kill
> LVS.

Is that a configuration that works for other people?

MasterDirector BackupDirector
| |
sync-master ------> sync-backup
| |
LocalNode(RS) |
^ |
| |
Client --> DR ---------> LocalNode(RS)

with no separate real servers? I hadn't ever tried that...

Things that might help diag the problem
What is your output from:
ipvsadm -L --daemon (on master and backup)
and
ipvsadm -l -n --stats
and
ipvsadm -S -n

HA config?
if/ip config?
LVS mngmt (ldirector/keepalived/etc) config?

Anything interesting in:
fgrep -i IPVS /var/log/messages

Can you verify that you can see the multicast packets coming from the master
on the backup director machine using tcpdump or equiv?

>
> I could move the LAN traffic off to another interface, and a crossover
> cable, but that really only moves the problem, and doesnt sort the
> CPU system issue.

Despite what Sal said, I would definitely do anything you can to separate sync
traffic from LVS Dir traffic because it will simplify your initial
troubleshooting - after you find the problem, you can go back to one interface
or whatever. You can use ifstat or my favorite dstat to monitor traffic per
interface - dstat can give you net interface details, CPU usage, etc all in a
single line. Then you can see where the bulk of the traffic is from/to.

With DR unless you have massive inbound traffic or huge overall traffic your
LVS Director to Real Server traffic should be relatively low.

>
>
> Best Regards
>
> Iain
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

stepedino at bhsoftware

Apr 5, 2007, 6:04 AM

Post #4 of 6 (580 views)
Permalink

Re: LVS DR increased "system CPU" and Excessive Network Traffic [In reply to]

On Wed, 2007-04-04 at 18:10 -0700, Rob wrote:
> Is that a configuration that works for other people?
> with no separate real servers? I hadn't ever tried that...

I'm running a LVS DR Localnode right now. Works fine.

> Despite what Sal said, I would definitely do anything you can to separate sync
> traffic from LVS Dir traffic because it will simplify your initial
> troubleshooting

Well, he was suggesting moving the traffic to another interface so it
can take up 20% of the interface and it wouldn't matter, and he wouldn't
have to deal with it, which I was saying, and he already figured, was
probably not a good idea because it just masks the problem and doesn't
really solve anything. It sounds like he's still setting things up, so
there's likely no other real traffic on this interface (except, perhaps,
ssh traffic from him logging in) so it might not be necessary to
separate it for troubleshooting, but it would help to be sure that all
the traffic he was seeing on whatever tools he's using was only from the
directors.

--
Sal Tepedino <stepedino [at] bhsoftware>
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

ipvsuser at itsbeen

Apr 5, 2007, 8:42 AM

Post #5 of 6 (572 views)
Permalink

Re: LVS DR increased "system CPU" and Excessive Network Traffic [In reply to]

Sal Tepedino wrote:
> On Wed, 2007-04-04 at 18:10 -0700, Rob wrote:
>> Is that a configuration that works for other people?
>> with no separate real servers? I hadn't ever tried that...
>
> I'm running a LVS DR Localnode right now. Works fine.

Sweet I will have to try that.

>
>> Despite what Sal said, I would definitely do anything you can to separate sync
>> traffic from LVS Dir traffic because it will simplify your initial
>> troubleshooting
>
> Well, he was suggesting moving the traffic to another interface so it
> can take up 20% of the interface and it wouldn't matter, and he wouldn't
> have to deal with it, which I was saying, and he already figured, was
> probably not a good idea because it just masks the problem and doesn't
> really solve anything. It sounds like he's still setting things up, so
> there's likely no other real traffic on this interface (except, perhaps,
> ssh traffic from him logging in) so it might not be necessary to
> separate it for troubleshooting, but it would help to be sure that all
> the traffic he was seeing on whatever tools he's using was only from the
> directors.

Ah, sorry, I should have been more clear - I agreed with you on the fact that
it wouldn't help with CPU utilization but didn't state that clearly.
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

graeme at graemef

Apr 5, 2007, 8:58 AM

Post #6 of 6 (595 views)
Permalink

Re: LVS DR increased "system CPU" and Excessive Network Traffic [In reply to]

On Wed, 2007-04-04 at 20:35 +0100, Iain Young wrote:
> I've just installed a couple of machines with LVS on, using the
> DR method. I also used the localnode feature, and started the
> sync daemon as master on one, and backup on the other.
>
> The nodes are functioning both as Director, and Real Servers,
> and now, both machines are:
>
> a) showing "System CPU" at around 20%, which vanishes if I kill
> LVS.
>
> b) Sending out an awful lot of LAN traffic (120,000 packets an
> hour!), which I'm guessing is multicast [which is to be expected],
> but surely it doesnt need *that* many packets! Again, it dissapears
> when I shutdown LVS.

For the moment, just try it without the sync daemon at all. What you're
probably seeing is packets being "reflected" between servers because of
the following hypothetical scenario:

1. New connection arrives on Server1, is handled by localnode on
Server1. Sync daemon sends established info to Server2. Connection gets
handled and cleared from LVS table.
2. Next new connection arrives on Server1 from same client, is
round-robinned to Server2, which has connection in its' table yet to
expire from Server1 and sends the packet back. Server1 looks up in its'
table and finds that it sent the packet to Server2, so sends it back
again, and so on, ad infinitum.

If dropping the sync daemon clears the problem, you need to investigate
using netfilter marks (fwmarks) on inbound connections and then
configure your LVS VSes to match mark values rather than IP addresses.
It helps if you have two NICs per server, as you can send the
inter-server traffic via the "backend" non-client facing LAN and
simplify the marking of packets quite significantly.

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads