Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Quagga: Dev

[RFC] AgentX support for Quagga

 

 

Quagga dev RSS feed   Index | Next | Previous | View Threaded


bernat at luffy

May 24, 2012, 1:56 AM

Post #1 of 19 (2310 views)
Permalink
[RFC] AgentX support for Quagga

Hi!

Here is a serie of patches adding AgentX support to Quagga.

The first patch is not related to AgentX support. It just fixes Quagge
compilation when using a separate build directory.

The second patch removes some cruft left from a migration from
UCD-SNMP (I think). This is just a small cleanup.

The third patch makes use of "net-snmp-config" in place of linking to
-lnetsnmp. It removes explicit link to -lcrypto. If needed, this will
be added by "net-snmp-config". "net-snmp-config" exists since at least
NetSNMP 4.2.

The fourth patch separates SMUX specific stuff from generic SNMP
stuff. smux.h now only contains the interface to SMUX for the
daemons. Everything else has been moved to smux.c. The idea is for
another SNMP implementation to provide the appropriate smux_*
functions. snmp.c is a new file containing oid_* functions (they are
independant of the SNMP implementation).

The last patch adds AgentX support by providing the appropriate smux_*
functions. This patch does not modify daemons. Because Net-SNMP hides
its internals, it is not possible to disable AgentX support once
enabled. This should not be a problem, because, like with SMUX, the
commands are not exported to vtysh and therefore, I suppose there is
no way to call "no agentx".

Also, the event loop is modified (in thread.c). NetSNMP hides the
appropriate file descriptors and timers.

The documentation has not been updated. This should be easy enough
once I am done with the other details.

The big problem now is traps. Currently, I do not send any trap with
AgentX. The SMUX implementation only sends SNMPv1 traps using the SMUX
peer OID as an enterprise OID. I think this is not the right way to do
it. Both OSPF-MIB (through OSPF-TRAP-MIB) and BGP-MIB defines
notifications. A notification is some kind of SNMPv2 TRAP. Let me
explain. With a SNMPv1 trap, you have: an enterprise OID, a generic
trap code (always 6 in our case) and a specific trap code. With a
notification, all those are replaced by a snmptrap OID.

It is possible to transform SNMPv1 trap to SNMPv2 trap and
vice-versa. However, if I just do this, I will continue to use
inappropriate enterprise OID (the SMUX peer OID). I should use the
appropriate notification objects, as defined in MIB.

My first proposition was to change smux_trap() signature to include
the appropriate enterprise OID (used to build snmp trap OID). Then, I
will also update the SMUX implementation. There are two drawbacks:

- daemons maintained outside of Quagga will also need to update (if
they use traps)
- people receiving traps will need to update their configuration to
use the correct OID instead of the previous ones.

Another proposition will be to just keep the SMUX peer OID. People may
change it to the appropriate value (since I will not use it as a SMUX
peer OID) to get correct snmp trap OID. For BGP and OSPF, this will
work because all traps are rooted at the same OID. I can however
provide a "smux_inform()" function allowing to override the enterprise
OID.

Any though? And any comment on the patches in general?

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


equinox at opensourcerouting

Jun 7, 2012, 9:52 AM

Post #2 of 19 (2259 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Thu, May 24, 2012 at 10:56:30AM +0200, Vincent Bernat wrote:
> Here is a serie of patches adding AgentX support to Quagga.

Unless someone objects, I'll pick all of these patches up. I'll queue
them for a day or two in a feature branch and then just merge it [we
don't really have any sensible testsuite for SNMP...]

Cheers and Thanks for your patches Vincent,

-David

> The first patch is not related to AgentX support. It just fixes Quagge
> compilation when using a separate build directory.
>
> The second patch removes some cruft left from a migration from
> UCD-SNMP (I think). This is just a small cleanup.
>
> The third patch makes use of "net-snmp-config" in place of linking to
> -lnetsnmp. It removes explicit link to -lcrypto. If needed, this will
> be added by "net-snmp-config". "net-snmp-config" exists since at least
> NetSNMP 4.2.
>
> The fourth patch separates SMUX specific stuff from generic SNMP
> stuff. smux.h now only contains the interface to SMUX for the
> daemons. Everything else has been moved to smux.c. The idea is for
> another SNMP implementation to provide the appropriate smux_*
> functions. snmp.c is a new file containing oid_* functions (they are
> independant of the SNMP implementation).
>
> The last patch adds AgentX support by providing the appropriate smux_*
> functions. This patch does not modify daemons. Because Net-SNMP hides
> its internals, it is not possible to disable AgentX support once
> enabled. This should not be a problem, because, like with SMUX, the
> commands are not exported to vtysh and therefore, I suppose there is
> no way to call "no agentx".
>
> Also, the event loop is modified (in thread.c). NetSNMP hides the
> appropriate file descriptors and timers.
>
> The documentation has not been updated. This should be easy enough
> once I am done with the other details.
>
> The big problem now is traps. Currently, I do not send any trap with
> AgentX. The SMUX implementation only sends SNMPv1 traps using the SMUX
> peer OID as an enterprise OID. I think this is not the right way to do
> it. Both OSPF-MIB (through OSPF-TRAP-MIB) and BGP-MIB defines
> notifications. A notification is some kind of SNMPv2 TRAP. Let me
> explain. With a SNMPv1 trap, you have: an enterprise OID, a generic
> trap code (always 6 in our case) and a specific trap code. With a
> notification, all those are replaced by a snmptrap OID.
>
> It is possible to transform SNMPv1 trap to SNMPv2 trap and
> vice-versa. However, if I just do this, I will continue to use
> inappropriate enterprise OID (the SMUX peer OID). I should use the
> appropriate notification objects, as defined in MIB.
>
> My first proposition was to change smux_trap() signature to include
> the appropriate enterprise OID (used to build snmp trap OID). Then, I
> will also update the SMUX implementation. There are two drawbacks:
>
> - daemons maintained outside of Quagga will also need to update (if
> they use traps)
> - people receiving traps will need to update their configuration to
> use the correct OID instead of the previous ones.
>
> Another proposition will be to just keep the SMUX peer OID. People may
> change it to the appropriate value (since I will not use it as a SMUX
> peer OID) to get correct snmp trap OID. For BGP and OSPF, this will
> work because all traps are rooted at the same OID. I can however
> provide a "smux_inform()" function allowing to override the enterprise
> OID.
>
> Any though? And any comment on the patches in general?
>
> _______________________________________________
> Quagga-dev mailing list
> Quagga-dev [at] lists
> http://lists.quagga.net/mailman/listinfo/quagga-dev
Attachments: signature.asc (0.22 KB)


bernat at luffy

Jun 7, 2012, 2:05 PM

Post #3 of 19 (2281 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

OoO Lors de la soirée naissante du jeudi 07 juin 2012, vers 18:52, David
Lamparter <equinox [at] opensourcerouting> disait :

>> Here is a serie of patches adding AgentX support to Quagga.

> Unless someone objects, I'll pick all of these patches up. I'll queue
> them for a day or two in a feature branch and then just merge it [we
> don't really have any sensible testsuite for SNMP...]

I will have an extensive deployment with those patches next week. Keep
them on hold and I will get back to you with the results. Also, be sure
to use the set from Github since I have done very small corrections
since the original posting and I didn't want to flood the mailing list
by resending the whole set of patches:
https://github.com/vincentbernat/quagga/tree/feature/agentx
https://github.com/vincentbernat/quagga/compare/master...feature/agentx

I can also resend the patches if you prefer. I also think that we should
add an entry in NEWS. Should I add a patch for this or is it only done
on merge? Something like this (non-native English speaker, feel free to
correct):

- [snmp] AgentX support has been added and is used by default. SMUX is
still available as a configure option. When using AgentX support,
notifications are rooted at the standard OID instead of at the OID
provided in the "smux peer" directive.
--
Vincent Bernat ☯ http://vincent.bernat.im

Watch out for off-by-one errors.
- The Elements of Programming Style (Kernighan & Plauger)

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


sodynet1 at gmail

Jun 7, 2012, 2:50 PM

Post #4 of 19 (2252 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

Hi,
updating the documentation of useage on quagga site is also preffered :)

Sami

On Fri, Jun 8, 2012 at 12:05 AM, Vincent Bernat <bernat [at] luffy> wrote:

> OoO Lors de la soirée naissante du jeudi 07 juin 2012, vers 18:52, David
> Lamparter <equinox [at] opensourcerouting> disait :
>
> >> Here is a serie of patches adding AgentX support to Quagga.
>
> > Unless someone objects, I'll pick all of these patches up. I'll queue
> > them for a day or two in a feature branch and then just merge it [.we
> > don't really have any sensible testsuite for SNMP...]
>
> I will have an extensive deployment with those patches next week. Keep
> them on hold and I will get back to you with the results. Also, be sure
> to use the set from Github since I have done very small corrections
> since the original posting and I didn't want to flood the mailing list
> by resending the whole set of patches:
> https://github.com/vincentbernat/quagga/tree/feature/agentx
> https://github.com/vincentbernat/quagga/compare/master...feature/agentx
>
> I can also resend the patches if you prefer. I also think that we should
> add an entry in NEWS. Should I add a patch for this or is it only done
> on merge? Something like this (non-native English speaker, feel free to
> correct):
>
> - [snmp] AgentX support has been added and is used by default. SMUX is
> still available as a configure option. When using AgentX support,
> notifications are rooted at the standard OID instead of at the OID
> provided in the "smux peer" directive.
> --
> Vincent Bernat ☯ http://vincent.bernat.im
>
> Watch out for off-by-one errors.
> - The Elements of Programming Style (Kernighan & Plauger)
>
> _______________________________________________
> Quagga-dev mailing list
> Quagga-dev [at] lists
> http://lists.quagga.net/mailman/listinfo/quagga-dev
>



--
Sami Halabi
Information Systems Engineer
NMS Projects Expert
FreeBSD Expert


bernat at luffy

Jun 7, 2012, 2:52 PM

Post #5 of 19 (2250 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

OoO La nuit ayant déjà recouvert d'encre ce jour du jeudi 07 juin 2012,
vers 23:50, Sami Halabi <sodynet1 [at] gmail> disait :

> Hi,
> updating the documentation of useage on quagga site is also preffered
> :)

There is a commit to update the documentation. I suppose that there will
be some automatic mirroring on the website.
--
Vincent Bernat ☯ http://vincent.bernat.im

#define BB_STAT2_TMP_INTR 0x10 /* My Penguins are burning.
Are you able to smell it? */
2.2.16 /usr/src/linux/include/asm-sparc/obio.h

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


bernat at luffy

Jun 17, 2012, 4:03 AM

Post #6 of 19 (2220 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

❦ 7 juin 2012 23:05 CEST, Vincent Bernat <bernat [at] luffy> :

>> Unless someone objects, I'll pick all of these patches up. I'll queue
>> them for a day or two in a feature branch and then just merge it [we
>> don't really have any sensible testsuite for SNMP...]
>
> I will have an extensive deployment with those patches next week. Keep
> them on hold and I will get back to you with the results. Also, be sure
> to use the set from Github since I have done very small corrections
> since the original posting and I didn't want to flood the mailing list
> by resending the whole set of patches:
> https://github.com/vincentbernat/quagga/tree/feature/agentx
> https://github.com/vincentbernat/quagga/compare/master...feature/agentx

Hi David!

The patchset has been tested on real servers for a week without any
problem. I have rewritten them to rebase them on top of master and to
respect GNU style. You'll find the full patchset here:

https://github.com/vincentbernat/quagga/compare/master...feature/agentx
https://github.com/vincentbernat/quagga.git (feature/agentx branch)
--
panic("CPU too expensive - making holiday in the ANDES!");
2.2.16 /usr/src/linux/arch/mips/kernel/traps.c

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


renatowestphal at gmail

Jun 25, 2012, 9:02 AM

Post #7 of 19 (2204 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

2012/5/24 Vincent Bernat <bernat [at] luffy>:
> Hi!
>
> Here is a serie of patches adding AgentX support to Quagga.
>
> The first patch is not related to AgentX support. It just fixes Quagge
> compilation when using a separate build directory.
>
> The second patch removes some cruft left from a migration from
> UCD-SNMP (I think). This is just a small cleanup.
>
> The third patch makes use of "net-snmp-config" in place of linking to
> -lnetsnmp. It removes explicit link to -lcrypto. If needed, this will
> be added by "net-snmp-config". "net-snmp-config" exists since at least
> NetSNMP 4.2.
>
> The fourth patch separates SMUX specific stuff from generic SNMP
> stuff. smux.h now only contains the interface to SMUX for the
> daemons. Everything else has been moved to smux.c. The idea is for
> another SNMP implementation to provide the appropriate smux_*
> functions. snmp.c is a new file containing oid_* functions (they are
> independant of the SNMP implementation).
>
> The last patch adds AgentX support by providing the appropriate smux_*
> functions. This patch does not modify daemons. Because Net-SNMP hides
> its internals, it is not possible to disable AgentX support once
> enabled. This should not be a problem, because, like with SMUX, the
> commands are not exported to vtysh and therefore, I suppose there is
> no way to call "no agentx".
>
> Also, the event loop is modified (in thread.c). NetSNMP hides the
> appropriate file descriptors and timers.
>
> The documentation has not been updated. This should be easy enough
> once I am done with the other details.
>
> The big problem now is traps. Currently, I do not send any trap with
> AgentX. The SMUX implementation only sends SNMPv1 traps using the SMUX
> peer OID as an enterprise OID. I think this is not the right way to do
> it. Both OSPF-MIB (through OSPF-TRAP-MIB) and BGP-MIB defines
> notifications. A notification is some kind of SNMPv2 TRAP. Let me
> explain. With a SNMPv1 trap, you have: an enterprise OID, a generic
> trap code (always 6 in our case) and a specific trap code. With a
> notification, all those are replaced by a snmptrap OID.
>
> It is possible to transform SNMPv1 trap to SNMPv2 trap and
> vice-versa. However, if I just do this, I will continue to use
> inappropriate enterprise OID (the SMUX peer OID). I should use the
> appropriate notification objects, as defined in MIB.
>
> My first proposition was to change smux_trap() signature to include
> the appropriate enterprise OID (used to build snmp trap OID). Then, I
> will also update the SMUX implementation. There are two drawbacks:
>
> - daemons maintained outside of Quagga will also need to update (if
> they use traps)
> - people receiving traps will need to update their configuration to
> use the correct OID instead of the previous ones.
>
> Another proposition will be to just keep the SMUX peer OID. People may
> change it to the appropriate value (since I will not use it as a SMUX
> peer OID) to get correct snmp trap OID. For BGP and OSPF, this will
> work because all traps are rooted at the same OID. I can however
> provide a "smux_inform()" function allowing to override the enterprise
> OID.
>
> Any though? And any comment on the patches in general?

Hi Vincent,

I have been recently playing with SNMP and I think Agentx support is a
huge improvement to Quagga.

It would be very nice if you could help me with these questions:
* Why it is not possible to build Quagga with both agentx and smux
support? Is there any special reason for this?
* Do you have any idea on how hard it would be to "upgrade" the BGP
code to use agentx instead of smux? The bgpd daemon has some
performance issues (bug 632) when doing a snmpwalk on a very large
table. I think agentx can help solving this problem.

Finally, your patches break the compilation of Quagga with smux
support. I forked* your repo on github and fixed it, please take a
look (some warnings introduced by your patches still need to be
fixed). I also upgraded the BGP MIB to the most recent one (RFC 4273).
Only some minor details have changed since the previous MIB. If
possible, I would like you to review my patch (well.. maybe I'm asking
too much already ;-)

* https://github.com/rwestphal/quagga-public/commits/vincent

Regards,
--
Renato Westphal

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


bernat at luffy

Jun 25, 2012, 10:17 AM

Post #8 of 19 (2200 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

❦ 25 juin 2012 18:02 CEST, Renato Westphal <renatowestphal [at] gmail> :

> It would be very nice if you could help me with these questions:
> * Why it is not possible to build Quagga with both agentx and smux
> support? Is there any special reason for this?

Yes, I would need an additional (thin) layer to register MIB to both
subsystems. I don't think this is worth the effort. SMUX is out-dated
and upgrading to AgentX is easy (just add "master agentx" in your
"snmpd.conf" and enable agentx in Quagga with "agentx").

> * Do you have any idea on how hard it would be to "upgrade" the BGP
> code to use agentx instead of smux? The bgpd daemon has some
> performance issues (bug 632) when doing a snmpwalk on a very large
> table. I think agentx can help solving this problem.

The upgrade is already done. The specific code to each daemon is
compatible (and already was) with both SMUX and AgentX (except the trap
part but I have already upgraded it). This is also why SMUX/AgentX is a
compile time option.

AgentX won't improve the situation from the performance point of view.

For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
of GETNEXT requests. For each GETNEXT request, we need to locate the
appropriate element to return and for this, we need to walk each route
table from the beginning.

An easy fix would be to remember where we were the last time we got a
GETNEXT. I will try to work on this. Is there some way to get a huge BGP
table (with exabgp maybe?)?

> Finally, your patches break the compilation of Quagga with smux
> support. I forked* your repo on github and fixed it, please take a
> look (some warnings introduced by your patches still need to be
> fixed).

You are right, I didn't retest SMUX when I added trap support. I have
added your patch and rebased my stack of patches (to make all patches
compile, since David did not merge them yet). I have therefore "broken"
your tree. You can get it right with something like (assuming the remote
name for my repository is "rvincent")

git fetch rvincent
git rebase --onto rvincent/feature/agentx vincent~1 vincent

(or you can just start a new branch and cherry-pick your last commit)

> I also upgraded the BGP MIB to the most recent one (RFC 4273). Only
> some minor details have changed since the previous MIB. If possible, I
> would like you to review my patch (well.. maybe I'm asking too much
> already ;-)

Your patch seems fine. I don't have a testbed to actually test it but I
don't see any problem with it.
--
Make it clear before you make it faster.
- The Elements of Programming Style (Kernighan & Plauger)


renatowestphal at gmail

Jun 26, 2012, 7:01 AM

Post #9 of 19 (2181 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

2012/6/25 Vincent Bernat <bernat [at] luffy>:
> The upgrade is already done. The specific code to each daemon is
> compatible (and already was) with both SMUX and AgentX (except the trap
> part but I have already upgraded it). This is also why SMUX/AgentX is a
> compile time option.

Thanks for the clarification. Everything makes sense for me now.

Just one note. I have to run Quagga (any daemon) with root privileges
in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
Failed to connect to the agentx master agent ([NIL]):".

> AgentX won't improve the situation from the performance point of view.
>
> For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
> of GETNEXT requests. For each GETNEXT request, we need to locate the
> appropriate element to return and for this, we need to walk each route
> table from the beginning.
>
> An easy fix would be to remember where we were the last time we got a
> GETNEXT.

I don't think that's the problem. I did some profiling with
callgrind[1] and found out that the libnetsnmp is inherently slow. I
did a snmpwalk over a table with 10k routes and only a small
percentage of the execution time (~ 1.3%) was to gather information
from the BGP table (in the bgp4PathAttrTable function). I might be
wrong, but I think there's nothing we can do about this problem.

[1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png

> I will try to work on this. Is there some way to get a huge BGP
> table (with exabgp maybe?)?

I'm using the bgp_simple.pl script. I followed this tutorial to get
started with it:
http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/

> You are right, I didn't retest SMUX when I added trap support. I have
> added your patch and rebased my stack of patches (to make all patches
> compile, since David did not merge them yet). I have therefore "broken"
> your tree. You can get it right with something like (assuming the remote
> name for my repository is "rvincent")
>
> git fetch rvincent
> git rebase --onto rvincent/feature/agentx vincent~1 vincent
>
> (or you can just start a new branch and cherry-pick your last commit)

Perfect. I hope your patches will be merged in a few days.

> Your patch seems fine. I don't have a testbed to actually test it but I
> don't see any problem with it.

Thanks for looking into it. I'll review this patch next week and then
send it to quagga-dev.

Regards,
--
Renato Westphal
_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


shemminger at vyatta

Jun 26, 2012, 8:06 AM

Post #10 of 19 (2184 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Tue, 26 Jun 2012 11:01:18 -0300
Renato Westphal <renatowestphal [at] gmail> wrote:

> 2012/6/25 Vincent Bernat <bernat [at] luffy>:
> > The upgrade is already done. The specific code to each daemon is
> > compatible (and already was) with both SMUX and AgentX (except the trap
> > part but I have already upgraded it). This is also why SMUX/AgentX is a
> > compile time option.
>
> Thanks for the clarification. Everything makes sense for me now.
>
> Just one note. I have to run Quagga (any daemon) with root privileges
> in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
> Failed to connect to the agentx master agent ([NIL]):".
>
> > AgentX won't improve the situation from the performance point of view.
> >
> > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
> > of GETNEXT requests. For each GETNEXT request, we need to locate the
> > appropriate element to return and for this, we need to walk each route
> > table from the beginning.
> >
> > An easy fix would be to remember where we were the last time we got a
> > GETNEXT.
>
> I don't think that's the problem. I did some profiling with
> callgrind[1] and found out that the libnetsnmp is inherently slow. I
> did a snmpwalk over a table with 10k routes and only a small
> percentage of the execution time (~ 1.3%) was to gather information
> from the BGP table (in the bgp4PathAttrTable function). I might be
> wrong, but I think there's nothing we can do about this problem.
>
> [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
>
> > I will try to work on this. Is there some way to get a huge BGP
> > table (with exabgp maybe?)?
>
> I'm using the bgp_simple.pl script. I followed this tutorial to get
> started with it:
> http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
>
> > You are right, I didn't retest SMUX when I added trap support. I have
> > added your patch and rebased my stack of patches (to make all patches
> > compile, since David did not merge them yet). I have therefore "broken"
> > your tree. You can get it right with something like (assuming the remote
> > name for my repository is "rvincent")
> >
> > git fetch rvincent
> > git rebase --onto rvincent/feature/agentx vincent~1 vincent
> >
> > (or you can just start a new branch and cherry-pick your last commit)
>
> Perfect. I hope your patches will be merged in a few days.
>
> > Your patch seems fine. I don't have a testbed to actually test it but I
> > don't see any problem with it.
>
> Thanks for looking into it. I'll review this patch next week and then
> send it to quagga-dev.
>
> Regards,

I did some work to speed up SNMP and it's handling of large tables.
A lot of the problem was N^2 insertion into tables, combined with cache
timeout being less than the load time.
_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


bernat at luffy

Jun 26, 2012, 10:35 AM

Post #11 of 19 (2198 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

❦ 26 juin 2012 16:01 CEST, Renato Westphal <renatowestphal [at] gmail> :

>> The upgrade is already done. The specific code to each daemon is
>> compatible (and already was) with both SMUX and AgentX (except the trap
>> part but I have already upgraded it). This is also why SMUX/AgentX is a
>> compile time option.
>
> Thanks for the clarification. Everything makes sense for me now.
>
> Just one note. I have to run Quagga (any daemon) with root privileges
> in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
> Failed to connect to the agentx master agent ([NIL]):".

Yes, you need to have the appropriate rights on the AgentX socket
(usually, /var/agentx/master). The setup could be done in
snmpd.conf with agentXPerms directive.

>
>> AgentX won't improve the situation from the performance point of view.
>>
>> For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
>> of GETNEXT requests. For each GETNEXT request, we need to locate the
>> appropriate element to return and for this, we need to walk each route
>> table from the beginning.
>>
>> An easy fix would be to remember where we were the last time we got a
>> GETNEXT.
>
> I don't think that's the problem. I did some profiling with
> callgrind[1] and found out that the libnetsnmp is inherently slow. I
> did a snmpwalk over a table with 10k routes and only a small
> percentage of the execution time (~ 1.3%) was to gather information
> from the BGP table (in the bgp4PathAttrTable function). I might be
> wrong, but I think there's nothing we can do about this problem.
>
> [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png

Then, maybe I am wrong. This needs more investigations.
--
Use the good features of a language; avoid the bad ones.
- The Elements of Programming Style (Kernighan & Plauger)

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


equinox at opensourcerouting

Jun 26, 2012, 10:46 AM

Post #12 of 19 (2170 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Tue, Jun 26, 2012 at 08:06:12AM -0700, Stephen Hemminger wrote:
> On Tue, 26 Jun 2012 11:01:18 -0300
> Renato Westphal <renatowestphal [at] gmail> wrote:
>
> > 2012/6/25 Vincent Bernat <bernat [at] luffy>:
> > > The upgrade is already done. The specific code to each daemon is
> > > compatible (and already was) with both SMUX and AgentX (except the trap
> > > part but I have already upgraded it). This is also why SMUX/AgentX is a
> > > compile time option.
> >
> > Thanks for the clarification. Everything makes sense for me now.
> >
> > Just one note. I have to run Quagga (any daemon) with root privileges
> > in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
> > Failed to connect to the agentx master agent ([NIL]):".
> >
> > > AgentX won't improve the situation from the performance point of view.
> > >
> > > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
> > > of GETNEXT requests. For each GETNEXT request, we need to locate the
> > > appropriate element to return and for this, we need to walk each route
> > > table from the beginning.
> > >
> > > An easy fix would be to remember where we were the last time we got a
> > > GETNEXT.
> >
> > I don't think that's the problem. I did some profiling with
> > callgrind[1] and found out that the libnetsnmp is inherently slow. I
> > did a snmpwalk over a table with 10k routes and only a small
> > percentage of the execution time (~ 1.3%) was to gather information
> > from the BGP table (in the bgp4PathAttrTable function). I might be
> > wrong, but I think there's nothing we can do about this problem.
> >
> > [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
> >
> > > I will try to work on this. Is there some way to get a huge BGP
> > > table (with exabgp maybe?)?
> >
> > I'm using the bgp_simple.pl script. I followed this tutorial to get
> > started with it:
> > http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
> >
> > > You are right, I didn't retest SMUX when I added trap support. I have
> > > added your patch and rebased my stack of patches (to make all patches
> > > compile, since David did not merge them yet). I have therefore "broken"
> > > your tree. You can get it right with something like (assuming the remote
> > > name for my repository is "rvincent")
> > >
> > > git fetch rvincent
> > > git rebase --onto rvincent/feature/agentx vincent~1 vincent
> > >
> > > (or you can just start a new branch and cherry-pick your last commit)
> >
> > Perfect. I hope your patches will be merged in a few days.
> >
> > > Your patch seems fine. I don't have a testbed to actually test it but I
> > > don't see any problem with it.
> >
> > Thanks for looking into it. I'll review this patch next week and then
> > send it to quagga-dev.
> >
> > Regards,
>
> I did some work to speed up SNMP and it's handling of large tables.
> A lot of the problem was N^2 insertion into tables, combined with cache
> timeout being less than the load time.

Do these changes conflict with Vincent's AgentX support? Could you
(re-)post them to the list?

Thanks in advance,

-David
Attachments: signature.asc (0.22 KB)


renatowestphal at gmail

Jun 26, 2012, 10:47 AM

Post #13 of 19 (2175 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

2012/6/26 David Lamparter <equinox [at] opensourcerouting>:
> On Tue, Jun 26, 2012 at 08:06:12AM -0700, Stephen Hemminger wrote:
>> On Tue, 26 Jun 2012 11:01:18 -0300
>> Renato Westphal <renatowestphal [at] gmail> wrote:
>>
>> > 2012/6/25 Vincent Bernat <bernat [at] luffy>:
>> > > The upgrade is already done. The specific code to each daemon is
>> > > compatible (and already was) with both SMUX and AgentX (except the trap
>> > > part but I have already upgraded it). This is also why SMUX/AgentX is a
>> > > compile time option.
>> >
>> > Thanks for the clarification. Everything makes sense for me now.
>> >
>> > Just one note. I have to run Quagga (any daemon) with root privileges
>> > in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
>> > Failed to connect to the agentx master agent ([NIL]):".
>> >
>> > > AgentX won't improve the situation from the performance point of view.
>> > >
>> > > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
>> > > of GETNEXT requests. For each GETNEXT request, we need to locate the
>> > > appropriate element to return and for this, we need to walk each route
>> > > table from the beginning.
>> > >
>> > > An easy fix would be to remember where we were the last time we got a
>> > > GETNEXT.
>> >
>> > I don't think that's the problem. I did some profiling with
>> > callgrind[1] and found out that the libnetsnmp is inherently slow. I
>> > did a snmpwalk over a table with 10k routes and only a small
>> > percentage of the execution time (~ 1.3%) was to gather information
>> > from the BGP table (in the bgp4PathAttrTable function). I might be
>> > wrong, but I think there's nothing we can do about this problem.
>> >
>> > [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
>> >
>> > > I will try to work on this. Is there some way to get a huge BGP
>> > > table (with exabgp maybe?)?
>> >
>> > I'm using the bgp_simple.pl script. I followed this tutorial to get
>> > started with it:
>> > http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
>> >
>> > > You are right, I didn't retest SMUX when I added trap support. I have
>> > > added your patch and rebased my stack of patches (to make all patches
>> > > compile, since David did not merge them yet). I have therefore "broken"
>> > > your tree. You can get it right with something like (assuming the remote
>> > > name for my repository is "rvincent")
>> > >
>> > > git fetch rvincent
>> > > git rebase --onto rvincent/feature/agentx vincent~1 vincent
>> > >
>> > > (or you can just start a new branch and cherry-pick your last commit)
>> >
>> > Perfect. I hope your patches will be merged in a few days.
>> >
>> > > Your patch seems fine. I don't have a testbed to actually test it but I
>> > > don't see any problem with it.
>> >
>> > Thanks for looking into it. I'll review this patch next week and then
>> > send it to quagga-dev.
>> >
>> > Regards,
>>
>> I did some work to speed up SNMP and it's handling of large tables.
>> A lot of the problem was N^2 insertion into tables, combined with cache
>> timeout being less than the load time.
>
> Do these changes conflict with Vincent's AgentX support? Could you
> (re-)post them to the list?
>
> Thanks in advance,
>
> -David

His changes are in the net-snmp, not quagga:
http://git.vyatta.com/git/?p=net-snmp.git;a=shortlog;h=refs/heads/oxnard

I'm testing them now.

--
Renato Westphal

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


shemminger at vyatta

Jun 26, 2012, 10:59 AM

Post #14 of 19 (2181 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Tue, 26 Jun 2012 14:47:52 -0300
Renato Westphal <renatowestphal [at] gmail> wrote:

> 2012/6/26 David Lamparter <equinox [at] opensourcerouting>:
> > On Tue, Jun 26, 2012 at 08:06:12AM -0700, Stephen Hemminger wrote:
> >> On Tue, 26 Jun 2012 11:01:18 -0300
> >> Renato Westphal <renatowestphal [at] gmail> wrote:
> >>
> >> > 2012/6/25 Vincent Bernat <bernat [at] luffy>:
> >> > > The upgrade is already done. The specific code to each daemon is
> >> > > compatible (and already was) with both SMUX and AgentX (except the trap
> >> > > part but I have already upgraded it). This is also why SMUX/AgentX is a
> >> > > compile time option.
> >> >
> >> > Thanks for the clarification. Everything makes sense for me now.
> >> >
> >> > Just one note. I have to run Quagga (any daemon) with root privileges
> >> > in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
> >> > Failed to connect to the agentx master agent ([NIL]):".
> >> >
> >> > > AgentX won't improve the situation from the performance point of view.
> >> > >
> >> > > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
> >> > > of GETNEXT requests. For each GETNEXT request, we need to locate the
> >> > > appropriate element to return and for this, we need to walk each route
> >> > > table from the beginning.
> >> > >
> >> > > An easy fix would be to remember where we were the last time we got a
> >> > > GETNEXT.
> >> >
> >> > I don't think that's the problem. I did some profiling with
> >> > callgrind[1] and found out that the libnetsnmp is inherently slow. I
> >> > did a snmpwalk over a table with 10k routes and only a small
> >> > percentage of the execution time (~ 1.3%) was to gather information
> >> > from the BGP table (in the bgp4PathAttrTable function). I might be
> >> > wrong, but I think there's nothing we can do about this problem.
> >> >
> >> > [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
> >> >
> >> > > I will try to work on this. Is there some way to get a huge BGP
> >> > > table (with exabgp maybe?)?
> >> >
> >> > I'm using the bgp_simple.pl script. I followed this tutorial to get
> >> > started with it:
> >> > http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
> >> >
> >> > > You are right, I didn't retest SMUX when I added trap support. I have
> >> > > added your patch and rebased my stack of patches (to make all patches
> >> > > compile, since David did not merge them yet). I have therefore "broken"
> >> > > your tree. You can get it right with something like (assuming the remote
> >> > > name for my repository is "rvincent")
> >> > >
> >> > > git fetch rvincent
> >> > > git rebase --onto rvincent/feature/agentx vincent~1 vincent
> >> > >
> >> > > (or you can just start a new branch and cherry-pick your last commit)
> >> >
> >> > Perfect. I hope your patches will be merged in a few days.
> >> >
> >> > > Your patch seems fine. I don't have a testbed to actually test it but I
> >> > > don't see any problem with it.
> >> >
> >> > Thanks for looking into it. I'll review this patch next week and then
> >> > send it to quagga-dev.
> >> >
> >> > Regards,
> >>
> >> I did some work to speed up SNMP and it's handling of large tables.
> >> A lot of the problem was N^2 insertion into tables, combined with cache
> >> timeout being less than the load time.
> >
> > Do these changes conflict with Vincent's AgentX support? Could you
> > (re-)post them to the list?
> >
> > Thanks in advance,
> >
> > -David
>
> His changes are in the net-snmp, not quagga:
> http://git.vyatta.com/git/?p=net-snmp.git;a=shortlog;h=refs/heads/oxnard
>
> I'm testing them now.
>

There is one more in our Vyatta version. Net-snmp project seems to be
one of those "hard to get patches accepted" projects so I stopped bothering

commit a4768cfe125f1481e05be8d2f0dece546b1083b6
Author: Stephen Hemminger <shemminger [at] vyatta>
Date: Mon Feb 20 12:11:15 2012 -0800

optimize insertion of binary array container

The binary array is one of the most common container types in net-snmp
but each insert causes a lookup and a resort of the whole array.
Instead, of sorting use the lookup process to determine the location
in the array to insert, and optimize for the most common case.

Most containers insert items in order, so it makes sense to optimize
for the case of appending. This especially important for huge tables
like the route table on a router.
---
snmplib/container_binary_array.c | 54 ++++++++++++++++++++++++++-----------
1 files changed, 38 insertions(+), 16 deletions(-)

diff --git a/snmplib/container_binary_array.c b/snmplib/container_binary_array.c
index 91298b3..5828670 100644
--- a/snmplib/container_binary_array.c
+++ b/snmplib/container_binary_array.c
@@ -36,7 +36,6 @@
typedef struct binary_array_table_s {
size_t max_size; /* Size of the current data table */
size_t count; /* Index of the next free entry */
- int dirty;
void **data; /* The table itself */
} binary_array_table;

@@ -48,75 +47,6 @@ typedef struct binary_array_iterator_s {

static netsnmp_iterator *_ba_iterator_get(netsnmp_container *c);

-/**********************************************************************
- *
- *
- *
- */
-static void
-array_qsort(void **data, int first, int last, netsnmp_container_compare *f)
-{
- int i, j;
- void *mid, *tmp;
-
- i = first;
- j = last;
- mid = data[first + ((last - first) >> 1)];
-
- do {
- while (i < last && (*f)(data[i], mid) < 0)
- ++i;
- while (j > first && (*f)(mid, data[j]) < 0)
- --j;
-
- if(i < j) {
- tmp = data[i];
- data[i] = data[j];
- data[j] = tmp;
- ++i;
- --j;
- }
- else if (i == j) {
- ++i;
- --j;
- break;
- }
- } while(i <= j);
-
- if (j > first)
- array_qsort(data, first, j, f);
-
- if (i < last)
- array_qsort(data, i, last, f);
-}
-
-static int
-Sort_Array(netsnmp_container *c)
-{
- binary_array_table *t = (binary_array_table*)c->container_data;
- netsnmp_assert(t!=NULL);
- netsnmp_assert(c->compare!=NULL);
-
- if (c->flags & CONTAINER_KEY_UNSORTED)
- return 0;
-
- if (t->dirty) {
- /*
- * Sort the table
- */
- if (t->count > 1)
- array_qsort(t->data, 0, t->count - 1, c->compare);
- t->dirty = 0;
-
- /*
- * no way to know if it actually changed... just assume so.
- */
- ++c->sync;
- }
-
- return 1;
-}
-
static int
linear_search(const void *val, netsnmp_container *c)
{
@@ -165,9 +95,6 @@ binary_search(const void *val, netsnmp_container *c, int exact)
return linear_search(val, c);
}

- if (t->dirty)
- Sort_Array(c);
-
while (len > 0) {
half = len >> 1;
middle = first;
@@ -219,7 +146,6 @@ netsnmp_binary_array_initialize(void)

t->max_size = 0;
t->count = 0;
- t->dirty = 0;
t->data = NULL;

return t;
@@ -275,12 +201,6 @@ netsnmp_binary_array_get(netsnmp_container *c, const void *key, int exact)
return NULL;

/*
- * if the table is dirty, sort it.
- */
- if (t->dirty)
- Sort_Array(c);
-
- /*
* if there is a key, search. Otherwise default is 0;
*/
if (key) {
@@ -343,12 +263,6 @@ netsnmp_binary_array_remove(netsnmp_container *c, const void *key, void **save)
return 0;

/*
- * if the table is dirty, sort it.
- */
- if (t->dirty)
- Sort_Array(c);
-
- /*
* search
*/
if ((index = binary_search(key, c, 1)) == -1)
@@ -365,9 +279,6 @@ netsnmp_binary_array_for_each(netsnmp_container *c,
binary_array_table *t = (binary_array_table*)c->container_data;
size_t i;

- if (sort && t->dirty)
- Sort_Array(c);
-
for (i = 0; i < t->count; ++i)
(*fe) (t->data[i], context);
}
@@ -387,7 +298,6 @@ netsnmp_binary_array_clear(netsnmp_container *c,
}

t->count = 0;
- t->dirty = 0;
++c->sync;
}

@@ -395,25 +305,51 @@ NETSNMP_STATIC_INLINE int
netsnmp_binary_array_insert(netsnmp_container *c, const void *entry)
{
binary_array_table *t = (binary_array_table*)c->container_data;
- int new_max, was_dirty = 0;
- void *new_data; /* Used for * a) extending the data table
- * * b) the next entry to use */
- /*
- * check for duplicates
- */
- if (! (c->flags & CONTAINER_KEY_ALLOW_DUPLICATES)) {
- was_dirty = t->dirty;
- new_data = netsnmp_binary_array_get(c, entry, 1);
- if (NULL != new_data) {
- DEBUGMSGTL(("container","not inserting duplicate key\n"));
- return -1;
+ int new_max;
+ int index = -1; /* append */
+
+ if (t->count == 0)
+ ; /* Trivial case of empty list */
+ else if (c->flags & CONTAINER_KEY_UNSORTED) {
+ /* Unsorted list, always append but check for dups */
+ if ( !(c->flags & CONTAINER_KEY_ALLOW_DUPLICATES) &&
+ linear_search(entry, c) != -1)
+ goto duplicate;
+
+ } else {
+ int last = t->count - 1;
+ int result;
+
+ /* Optimize for case of appending to end of array. */
+ if ( (result = c->compare(t->data[last], entry)) <= 0) {
+ /* and check for duplicate */
+ if ( !(c->flags & CONTAINER_KEY_ALLOW_DUPLICATES) &&
+ result == 0)
+ goto duplicate;
+ } else {
+ /* Find nearest match.
+ * Returns the index of the entry to put after this one.
+ * -1 means put at the end.
+ */
+ index = binary_search(entry, c, 0);
+
+ /*
+ * If key is duplicate then the entry before it
+ * will be the same.
+ */
+ if ( !(c->flags & CONTAINER_KEY_ALLOW_DUPLICATES) &&
+ index > 0 &&
+ c->compare(t->data[index - 1], entry) == 0)
+ goto duplicate;
}
}
-
+
/*
* check if we need to resize the array
*/
if (t->max_size <= t->count) {
+ void **new_data;
+
/*
* Table is full, so extend it to double the size
*/
@@ -425,25 +361,29 @@ netsnmp_binary_array_insert(netsnmp_container *c, const void *entry)
if (new_data == NULL)
return -1;

- t->data = (void**)new_data;
+ t->data = new_data;
t->max_size = new_max;
}

/*
* Insert the new entry into the data array
*/
- t->data[t->count++] = NETSNMP_REMOVE_CONST(void *, entry);
- t->dirty = 1;
+ if (index == -1)
+ index = t->count;
+ else
+ memmove(&t->data[index+1], &t->data[index],
+ sizeof(void *) * (t->count - index));

- /*
- * if array was dirty before we called get, sync was incremented when
- * get called SortArray. If we didn't call get or the array wasn't dirty,
- * bump sync now.
- */
- if (!was_dirty)
- ++c->sync;
+ t->data[index] = NETSNMP_REMOVE_CONST(void *, entry);
+ t->count++;
+
+ ++c->sync;

return 0;
+
+duplicate:
+ DEBUGMSGTL(("container","not inserting duplicate key\n"));
+ return -1;
}

/**********************************************************************
@@ -464,9 +404,6 @@ binary_search_for_start(netsnmp_index *val, netsnmp_container *c)
if (!len)
return -1;

- if (t->dirty)
- Sort_Array(c);
-
while (len > 0) {
half = len >> 1;
middle = first;
@@ -506,12 +443,6 @@ netsnmp_binary_array_get_subset(netsnmp_container *c, void *key, int *len)
return NULL;

/*
- * if the table is dirty, sort it.
- */
- if (t->dirty)
- Sort_Array(c);
-
- /*
* find matching items
*/
start = end = binary_search_for_start((netsnmp_index *)key, c);
@@ -649,7 +580,6 @@ _ba_duplicate(netsnmp_container *c, void *ctx, u_int flags)

dupt->max_size = t->max_size;
dupt->count = t->count;
- dupt->dirty = t->dirty;

/*
* shallow copy
@@ -831,9 +761,6 @@ _ba_iterator_reset(binary_array_iterator *it)
return -1;
}

- if (t->dirty)
- Sort_Array(it->base.container);
-
/*
* save sync count, to make sure container doesn't change while
* iterator is in use.

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


equinox at opensourcerouting

Jun 26, 2012, 11:01 AM

Post #15 of 19 (2180 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Tue, Jun 26, 2012 at 02:47:52PM -0300, Renato Westphal wrote:
> 2012/6/26 David Lamparter <equinox [at] opensourcerouting>:
> > On Tue, Jun 26, 2012 at 08:06:12AM -0700, Stephen Hemminger wrote:
> >> On Tue, 26 Jun 2012 11:01:18 -0300
> >> Renato Westphal <renatowestphal [at] gmail> wrote:
> >>
> >> > 2012/6/25 Vincent Bernat <bernat [at] luffy>:
> >> > > The upgrade is already done. The specific code to each daemon is
> >> > > compatible (and already was) with both SMUX and AgentX (except the trap
> >> > > part but I have already upgraded it). This is also why SMUX/AgentX is a
> >> > > compile time option.
> >> >
> >> > Thanks for the clarification. Everything makes sense for me now.
> >> >
> >> > Just one note. I have to run Quagga (any daemon) with root privileges
> >> > in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
> >> > Failed to connect to the agentx master agent ([NIL]):".
> >> >
> >> > > AgentX won't improve the situation from the performance point of view.
> >> > >
> >> > > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
> >> > > of GETNEXT requests. For each GETNEXT request, we need to locate the
> >> > > appropriate element to return and for this, we need to walk each route
> >> > > table from the beginning.
> >> > >
> >> > > An easy fix would be to remember where we were the last time we got a
> >> > > GETNEXT.
> >> >
> >> > I don't think that's the problem. I did some profiling with
> >> > callgrind[1] and found out that the libnetsnmp is inherently slow. I
> >> > did a snmpwalk over a table with 10k routes and only a small
> >> > percentage of the execution time (~ 1.3%) was to gather information
> >> > from the BGP table (in the bgp4PathAttrTable function). I might be
> >> > wrong, but I think there's nothing we can do about this problem.
> >> >
> >> > [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
> >> >
> >> > > I will try to work on this. Is there some way to get a huge BGP
> >> > > table (with exabgp maybe?)?
> >> >
> >> > I'm using the bgp_simple.pl script. I followed this tutorial to get
> >> > started with it:
> >> > http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
> >> >
> >> > > You are right, I didn't retest SMUX when I added trap support. I have
> >> > > added your patch and rebased my stack of patches (to make all patches
> >> > > compile, since David did not merge them yet). I have therefore "broken"
> >> > > your tree. You can get it right with something like (assuming the remote
> >> > > name for my repository is "rvincent")
> >> > >
> >> > > git fetch rvincent
> >> > > git rebase --onto rvincent/feature/agentx vincent~1 vincent
> >> > >
> >> > > (or you can just start a new branch and cherry-pick your last commit)
> >> >
> >> > Perfect. I hope your patches will be merged in a few days.
> >> >
> >> > > Your patch seems fine. I don't have a testbed to actually test it but I
> >> > > don't see any problem with it.
> >> >
> >> > Thanks for looking into it. I'll review this patch next week and then
> >> > send it to quagga-dev.
> >> >
> >> > Regards,
> >>
> >> I did some work to speed up SNMP and it's handling of large tables.
> >> A lot of the problem was N^2 insertion into tables, combined with cache
> >> timeout being less than the load time.
> >
> > Do these changes conflict with Vincent's AgentX support?  Could you
> > (re-)post them to the list?
> >
> > Thanks in advance,
> >
> > -David
>
> His changes are in the net-snmp, not quagga:
> http://git.vyatta.com/git/?p=net-snmp.git;a=shortlog;h=refs/heads/oxnard

Oh, ah.

> I'm testing them now.

Perfect, thanks!
Attachments: signature.asc (0.22 KB)


renatowestphal at gmail

Jun 26, 2012, 1:36 PM

Post #16 of 19 (2176 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

2012/6/26 David Lamparter <equinox [at] opensourcerouting>:
> On Tue, Jun 26, 2012 at 02:47:52PM -0300, Renato Westphal wrote:
>> 2012/6/26 David Lamparter <equinox [at] opensourcerouting>:
>> > On Tue, Jun 26, 2012 at 08:06:12AM -0700, Stephen Hemminger wrote:
>> >> On Tue, 26 Jun 2012 11:01:18 -0300
>> >> Renato Westphal <renatowestphal [at] gmail> wrote:
>> >>
>> >> > 2012/6/25 Vincent Bernat <bernat [at] luffy>:
>> >> > > The upgrade is already done. The specific code to each daemon is
>> >> > > compatible (and already was) with both SMUX and AgentX (except the trap
>> >> > > part but I have already upgraded it). This is also why SMUX/AgentX is a
>> >> > > compile time option.
>> >> >
>> >> > Thanks for the clarification. Everything makes sense for me now.
>> >> >
>> >> > Just one note. I have to run Quagga (any daemon) with root privileges
>> >> > in order to use agentx. Otherwise I get an "snmp[warning]: Warning:
>> >> > Failed to connect to the agentx master agent ([NIL]):".
>> >> >
>> >> > > AgentX won't improve the situation from the performance point of view.
>> >> > >
>> >> > > For bgpd, I suspect the problem to be fairly simple: snmpwalk is a serie
>> >> > > of GETNEXT requests. For each GETNEXT request, we need to locate the
>> >> > > appropriate element to return and for this, we need to walk each route
>> >> > > table from the beginning.
>> >> > >
>> >> > > An easy fix would be to remember where we were the last time we got a
>> >> > > GETNEXT.
>> >> >
>> >> > I don't think that's the problem. I did some profiling with
>> >> > callgrind[1] and found out that the libnetsnmp is inherently slow. I
>> >> > did a snmpwalk over a table with 10k routes and only a small
>> >> > percentage of the execution time (~ 1.3%) was to gather information
>> >> > from the BGP table (in the bgp4PathAttrTable function). I might be
>> >> > wrong, but I think there's nothing we can do about this problem.
>> >> >
>> >> > [1] http://www.inf.ufrgs.br/~rwestphal/files/callgrind.png
>> >> >
>> >> > > I will try to work on this. Is there some way to get a huge BGP
>> >> > > table (with exabgp maybe?)?
>> >> >
>> >> > I'm using the bgp_simple.pl script. I followed this tutorial to get
>> >> > started with it:
>> >> > http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
>> >> >
>> >> > > You are right, I didn't retest SMUX when I added trap support. I have
>> >> > > added your patch and rebased my stack of patches (to make all patches
>> >> > > compile, since David did not merge them yet). I have therefore "broken"
>> >> > > your tree. You can get it right with something like (assuming the remote
>> >> > > name for my repository is "rvincent")
>> >> > >
>> >> > > git fetch rvincent
>> >> > > git rebase --onto rvincent/feature/agentx vincent~1 vincent
>> >> > >
>> >> > > (or you can just start a new branch and cherry-pick your last commit)
>> >> >
>> >> > Perfect. I hope your patches will be merged in a few days.
>> >> >
>> >> > > Your patch seems fine. I don't have a testbed to actually test it but I
>> >> > > don't see any problem with it.
>> >> >
>> >> > Thanks for looking into it. I'll review this patch next week and then
>> >> > send it to quagga-dev.
>> >> >
>> >> > Regards,
>> >>
>> >> I did some work to speed up SNMP and it's handling of large tables.
>> >> A lot of the problem was N^2 insertion into tables, combined with cache
>> >> timeout being less than the load time.
>> >
>> > Do these changes conflict with Vincent's AgentX support? Could you
>> > (re-)post them to the list?
>> >
>> > Thanks in advance,
>> >
>> > -David
>>
>> His changes are in the net-snmp, not quagga:
>> http://git.vyatta.com/git/?p=net-snmp.git;a=shortlog;h=refs/heads/oxnard
>
> Oh, ah.
>
>> I'm testing them now.
>
> Perfect, thanks!

Well, I don't know if I did something wrong but I didn't notice any
performance improvements after applying Stephen's patches on net-snmp.
I'll take a better look at it tomorrow..

--
Renato Westphal

_______________________________________________
Quagga-dev mailing list
Quagga-dev [at] lists
http://lists.quagga.net/mailman/listinfo/quagga-dev


equinox at opensourcerouting

Jul 3, 2012, 8:59 AM

Post #17 of 19 (2115 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Thu, May 24, 2012 at 10:56:30AM +0200, Vincent Bernat wrote:
> Here is a serie of patches adding AgentX support to Quagga.

Merging this now, if anyone had a problem with it they had long enough
to complain :)

-David
Attachments: signature.asc (0.22 KB)


bernat at luffy

Jul 10, 2012, 12:44 AM

Post #18 of 19 (2067 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

❦ 3 juillet 2012 17:59 CEST, David Lamparter <equinox [at] opensourcerouting> :

> On Thu, May 24, 2012 at 10:56:30AM +0200, Vincent Bernat wrote:
>> Here is a serie of patches adding AgentX support to Quagga.
>
> Merging this now, if anyone had a problem with it they had long enough
> to complain :)

Hi David!

Where did you merge the changes? I don't find anything on the master
branch. Did you also merge the OSPFv3 part? I have the following patches
that applies to this part. ospf6d was segfaulting when requesting
inexistant interfaces or inexistant areas.
Attachments: 0001-ospf6d-fix-segfault-when-requesting-inexistant-inter.patch (1.96 KB)


equinox at opensourcerouting

Jul 11, 2012, 2:26 AM

Post #19 of 19 (2064 views)
Permalink
Re: [RFC] AgentX support for Quagga [In reply to]

On Tue, Jul 10, 2012 at 09:44:34AM +0200, Vincent Bernat wrote:
> ❦ 3 juillet 2012 17:59 CEST, David Lamparter <equinox [at] opensourcerouting> :
>
> > On Thu, May 24, 2012 at 10:56:30AM +0200, Vincent Bernat wrote:
> >> Here is a serie of patches adding AgentX support to Quagga.
> >
> > Merging this now, if anyone had a problem with it they had long enough
> > to complain :)
>
> Hi David!
>
> Where did you merge the changes? I don't find anything on the master
> branch. Did you also merge the OSPFv3 part? I have the following patches
> that applies to this part. ospf6d was segfaulting when requesting
> inexistant interfaces or inexistant areas.

Sorry, I'm having some health issues that are delaying me. It'll be up
on master/code.quagga.net as soon as I can get rid of my migraine
without ending up in a hazy half-coma...

I was planning to pick up the OSPFv3 part too.


-David
Attachments: signature.asc (0.22 KB)

Quagga dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.