Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] net-snmp-lvs-module: stats fail under high load

  

 
Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


robin-lists at robinbowes

Nov 25, 2009, 5:22 AM

Post #1 of 8 (1474 views)
Permalink
[lvs-users] net-snmp-lvs-module: stats fail under high load
Hi,

I've been building out a platform that uses the net-snmp-lvs module to
monitor LVS stats. In testing, it worked just fine. I'm graphing
connection rate, packet rate, and byte rate for the virtual service and
each of the real servers.

However, when we put some live traffic through the LVS instance this
morning, we lost the connection rate stats. There are some graphs here
that show the issue:

http://yo61.com/net-snmp-lvs-module-fails-under-high-load.html

Sadly, I didn't think to try ipvsadm --list --rate at the time to see if
the problem was in ipvs or the net-snmp-lvs module and we've reverted
the cut-over so the system is no longer heavily loaded.

Has anyone else seen anything similar to this?

Thanks,

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


robin-lists at robinbowes

Dec 8, 2009, 4:49 AM

Post #2 of 8 (1377 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On 25/11/09 13:22, Robin Bowes wrote:
> Hi,
>
> I've been building out a platform that uses the net-snmp-lvs module to
> monitor LVS stats. In testing, it worked just fine. I'm graphing
> connection rate, packet rate, and byte rate for the virtual service and
> each of the real servers.
>
> However, when we put some live traffic through the LVS instance this
> morning, we lost the connection rate stats. There are some graphs here
> that show the issue:
>
> http://yo61.com/net-snmp-lvs-module-fails-under-high-load.html
>
> Sadly, I didn't think to try ipvsadm --list --rate at the time to see if
> the problem was in ipvs or the net-snmp-lvs module and we've reverted
> the cut-over so the system is no longer heavily loaded.
>
> Has anyone else seen anything similar to this?

*bump*

No-one got any thoughts on this?

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Dec 8, 2009, 6:20 AM

Post #3 of 8 (1392 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On Tue, 2009-12-08 at 12:49 +0000, Robin Bowes wrote:
> No-one got any thoughts on this?

I'd hazard a guess that the counters wrapped once (or more) inside your
polling period. This is a common problem when collecting counter-type
data which wraps.

Not having any experience of the SNMP module, can you tell me:

1. Is the counter 32 or 64 bit?
2. What is your polling period?

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


robin-lists at robinbowes

Dec 8, 2009, 8:27 AM

Post #4 of 8 (1377 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On 08/12/09 14:20, Graeme Fowler wrote:
> On Tue, 2009-12-08 at 12:49 +0000, Robin Bowes wrote:
>> No-one got any thoughts on this?
>
> I'd hazard a guess that the counters wrapped once (or more) inside your
> polling period. This is a common problem when collecting counter-type
> data which wraps.
>
> Not having any experience of the SNMP module, can you tell me:
>
> 1. Is the counter 32 or 64 bit?

Graeme,

Thanks for the reply.

Looking in the MIB, I see the following definitions:

LvsServiceEntry ::= SEQUENCE {
lvsServiceNumber INTEGER,
lvsServiceSchedType OCTET STRING,
lvsServiceProto INTEGER,
lvsServiceAddr IpAddress,
lvsServicePort INTEGER,
lvsServiceFWMark INTEGER,
lvsServicePersistTimeout TimeInterval,
lvsServiceNumDests INTEGER,
lvsServiceStatsConns Counter32,
lvsServiceStatsInPkts Counter32,
lvsServiceStatsOutPkts Counter32,
lvsServiceStatsInBytes Counter64,
lvsServiceStatsOutBytes Counter64,
lvsServiceRateCps Gauge32,
lvsServiceRateInPPS Gauge32,
lvsServiceRateOutPPS Gauge32,
lvsServiceRateInBPS Gauge32,
lvsServiceRateOutBPS Gauge32
}


So, looks like that might be the problem; lvsServiceStatsConns is 32-bit.

I wonder if it's as simple as changing ASN_COUNTER to ASN_COUNTER64 in
the appropriate place in lvs.c ?

> 2. What is your polling period?

I think I'm polling at 5 min intervales.

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Dec 8, 2009, 10:07 AM

Post #5 of 8 (1382 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On Tue, 2009-12-08 at 16:27 +0000, Robin Bowes wrote:
> So, looks like that might be the problem; lvsServiceStatsConns is 32-bit.

I thought so.

> I wonder if it's as simple as changing ASN_COUNTER to ASN_COUNTER64 in
> the appropriate place in lvs.c ?

Hrm... this is in the SNMP module code, right? Or do you mean in the
main LVS code?

> > 2. What is your polling period?
>
> I think I'm polling at 5 min intervales.

Take note of the following page:

http://rrdtool.mirror.graemef.net/doc/rrdcreate.en.html

Have a read of "NOTE on COUNTER vs DERIVE". You could, in theory, change
your RRD definitions to use DERIVE with a min of 0 and poll every
minute, which might get the data you need.

Alternatively, if the code change is inside the SNMP module code, you
could try it but you may have to change the OID that's output as the MIB
is fairly well-defined already.

Graeme





_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


robin-lists at robinbowes

Dec 8, 2009, 10:09 AM

Post #6 of 8 (1392 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On 08/12/09 16:27, Robin Bowes wrote:
> So, looks like that might be the problem; lvsServiceStatsConns is 32-bit.

Wrong!

> I wonder if it's as simple as changing ASN_COUNTER to ASN_COUNTER64 in
> the appropriate place in lvs.c ?

Well, I don't think it will hurt, but it didn't fix things.

I tried using ipvsadm while throwing lots of connections at the LVS host:

# ipvsadm --list --rate
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port CPS InPPS OutPPS InBPS
OutBPS
-> RemoteAddress:Port
TCP 192.168.55.60:http 0 52619 0 3578025
0
-> b011.private.b.example.com 0 6580 0 447439
0
-> b013.private.b.example.com 0 6578 0 447290
0
-> b012.private.b.example.com 0 6581 0 447528
0
-> b010.private.b.example.com 0 6573 0 446936
0
-> b008.private.b.example.com 0 6581 0 447507
0
-> b009.private.b.example.com 0 6578 0 447330
0
-> b007.private.b.example.com 0 6573 0 446939
0
-> b006.private.b.example.com 0 6574 0 447056
0

Notice that ipvsadm reports "02 for connection rate too, so it looks
like the problem is in (lib)ipvsadm.

Now, net-snmp-lvs-module-0.0.4 comes with ipvsadm v1.24 and I see v1.25
is available. However, I've failed to get net-snmp-lvs-module to build
with ipvsadm 1.25 on CentOS 5.4

make -C libipvs
make[1]: Entering directory
`/home/rpmbuild/rpmbuild/BUILD/net-snmp-lvs-module-0.0.4/ipvsadm-1.25/libipvs'
gcc -Wall -Wunused -Wstrict-prototypes -g -fPIC -DLIBIPVS_USE_NL
-DHAVE_NET_IP_VS_H -c -o libipvs.o libipvs.c
In file included from libipvs.h:13,
from libipvs.c:23:
ip_vs.h:16:31: error: netlink/genl/genl.h: No such file or directory
ip_vs.h:17:31: error: netlink/genl/ctrl.h: No such file or directory
In file included from libipvs.h:13,
from libipvs.c:23:

I've installed libnl-devel, but it doesn't contain those files in v1.0.
I've also tried building libnl v1.1 but that won't build on CentOS either:

rpmbuild -ba libnl.spec
[snip]
gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -g
-I./include -I../include -I. -D_GNU_SOURCE -Wall -ggdb -fPIC -c -o
route/link/vlan.o route/link/vlan.c
route/link/vlan.c:51: error: 'VLAN_FLAG_REORDER_HDR' undeclared here
(not in a function)
make[2]: *** [route/link/vlan.o] Error 1
gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -g
-I./include -I../include -I. -D_GNU_SOURCE -Wall -ggdb -fPIC -c -o
route/link/vlan.o route/link/vlan.c
route/link/vlan.c:51: error: 'VLAN_FLAG_REORDER_HDR' undeclared here
(not in a function)
make[2]: *** [route/link/vlan.o] Error 1
make[1]: *** [all] Error 2
make: *** [all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.32736 (%build)

Any ideas how I might take this forward?

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


robin-lists at robinbowes

Dec 8, 2009, 10:21 AM

Post #7 of 8 (1376 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On 08/12/09 18:09, Robin Bowes wrote:
> Notice that ipvsadm reports "02 for connection rate too,

That should, of course, be "0".

I guess I need to raise a bug about the ipvsadm issue.

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


robin-lists at robinbowes

Dec 9, 2009, 3:44 AM

Post #8 of 8 (1356 views)
Permalink
Re: [lvs-users] net-snmp-lvs-module: stats fail under high load [In reply to]
On 08/12/09 18:21, Robin Bowes wrote:

> I guess I need to raise a bug about the ipvsadm issue.


https://bugzilla.redhat.com/show_bug.cgi?id=545793

R.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.