Mailing List Archive: Cisco: NSP

Tuning HSRP timers for BGP routers

Index | Next | Previous | View Threaded

mhuff at ox

May 9, 2012, 6:45 AM

Post #1 of 10 (936 views)
Permalink

Tuning HSRP timers for BGP routers

We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are using
default timers for the HSRP interfaces, and we are seeing nightly HSRP state
changes. Not a lot, but 1-2 a night. This appears to only have started
recently. We are looking at logs, but I assume it's due to BGP cpu
exhaustion. We don't see any L2 errors on the VLAN where the HSRP is
running, so I don't think it's a physical problem.

What timers do people use for HSRP on BGP routers as a practice? Obviously
we want the smallest timers that would be possible.

----

Matthew Huff | 1 Manhattanville Rd

Director of Operations | Purchase, NY 10577

OTA Management LLC | Phone: 914-460-4039

aim: matthewbhuff | Fax: 914-460-4139

Attachments:

smime.p7s (5.21 KB)

mack.mcbride at viawest

May 9, 2012, 10:38 AM

Post #2 of 10 (860 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

We use timers 2 and 6 on multiple sets of 6500s with a relatively large number of VLANs.
The 6500 has a much less powerful CPU and seems to handle it ok.
However, the 7200 is a software device so through traffic can have a major effect on CPU.
It may be you have large high BW bursts during backups or something similar.
You should probably do some correlation analysis before blaming the BGP.
If you are getting that kind of BGP event on a nightly basis you should probably talk to your provider.

LR Mack McBride
Network Architect

-----Original Message-----
From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-bounces [at] puck] On Behalf Of Matthew Huff
Sent: Wednesday, May 09, 2012 7:46 AM
To: 'cisco-nsp [at] puck'
Subject: [c-nsp] Tuning HSRP timers for BGP routers

We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are using default timers for the HSRP interfaces, and we are seeing nightly HSRP state changes. Not a lot, but 1-2 a night. This appears to only have started recently. We are looking at logs, but I assume it's due to BGP cpu exhaustion. We don't see any L2 errors on the VLAN where the HSRP is running, so I don't think it's a physical problem.

What timers do people use for HSRP on BGP routers as a practice? Obviously we want the smallest timers that would be possible.

----

Matthew Huff | 1 Manhattanville Rd

Director of Operations | Purchase, NY 10577

OTA Management LLC | Phone: 914-460-4039

aim: matthewbhuff | Fax: 914-460-4139

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

mhuff at ox

May 9, 2012, 11:21 AM

Post #3 of 10 (866 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

That's the thing, we aren't getting any BGP events, just HSRP ones. The
netflow analysis don't show a high bandwidth utilization, that's why I was
looking at the BGP re-calc causing the HSRP hellos to drop.

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff | Fax: 914-460-4139

> -----Original Message-----
> From: Mack McBride [mailto:mack.mcbride [at] viawest]
> Sent: Wednesday, May 09, 2012 1:39 PM
> To: Matthew Huff; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> We use timers 2 and 6 on multiple sets of 6500s with a relatively large
> number of VLANs.
> The 6500 has a much less powerful CPU and seems to handle it ok.
> However, the 7200 is a software device so through traffic can have a
> major effect on CPU.
> It may be you have large high BW bursts during backups or something
> similar.
> You should probably do some correlation analysis before blaming the
> BGP.
> If you are getting that kind of BGP event on a nightly basis you
> should probably talk to your provider.
>
> LR Mack McBride
> Network Architect
>
> -----Original Message-----
> From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-
> bounces [at] puck] On Behalf Of Matthew Huff
> Sent: Wednesday, May 09, 2012 7:46 AM
> To: 'cisco-nsp [at] puck'
> Subject: [c-nsp] Tuning HSRP timers for BGP routers
>
> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
> using default timers for the HSRP interfaces, and we are seeing nightly
> HSRP state changes. Not a lot, but 1-2 a night. This appears to only
> have started recently. We are looking at logs, but I assume it's due
> to BGP cpu exhaustion. We don't see any L2 errors on the VLAN where the
> HSRP is running, so I don't think it's a physical problem.
>
>
>
> What timers do people use for HSRP on BGP routers as a practice?
> Obviously we want the smallest timers that would be possible.
>
>
>
> ----
>
> Matthew Huff | 1 Manhattanville Rd
>
> Director of Operations | Purchase, NY 10577
>
> OTA Management LLC | Phone: 914-460-4039
>
> aim: matthewbhuff | Fax: 914-460-4139
>
>

Attachments:

smime.p7s (5.21 KB)

mack.mcbride at viawest

May 9, 2012, 11:23 AM

Post #4 of 10 (873 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

What is your base CPU utilization?

Mack

-----Original Message-----
From: Matthew Huff [mailto:mhuff [at] ox]
Sent: Wednesday, May 09, 2012 12:22 PM
To: Mack McBride; 'cisco-nsp [at] puck'
Subject: RE: Tuning HSRP timers for BGP routers

That's the thing, we aren't getting any BGP events, just HSRP ones. The
netflow analysis don't show a high bandwidth utilization, that's why I was
looking at the BGP re-calc causing the HSRP hellos to drop.

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff | Fax: 914-460-4139

> -----Original Message-----
> From: Mack McBride [mailto:mack.mcbride [at] viawest]
> Sent: Wednesday, May 09, 2012 1:39 PM
> To: Matthew Huff; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> We use timers 2 and 6 on multiple sets of 6500s with a relatively large
> number of VLANs.
> The 6500 has a much less powerful CPU and seems to handle it ok.
> However, the 7200 is a software device so through traffic can have a
> major effect on CPU.
> It may be you have large high BW bursts during backups or something
> similar.
> You should probably do some correlation analysis before blaming the
> BGP.
> If you are getting that kind of BGP event on a nightly basis you
> should probably talk to your provider.
>
> LR Mack McBride
> Network Architect
>
> -----Original Message-----
> From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-
> bounces [at] puck] On Behalf Of Matthew Huff
> Sent: Wednesday, May 09, 2012 7:46 AM
> To: 'cisco-nsp [at] puck'
> Subject: [c-nsp] Tuning HSRP timers for BGP routers
>
> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
> using default timers for the HSRP interfaces, and we are seeing nightly
> HSRP state changes. Not a lot, but 1-2 a night. This appears to only
> have started recently. We are looking at logs, but I assume it's due
> to BGP cpu exhaustion. We don't see any L2 errors on the VLAN where the
> HSRP is running, so I don't think it's a physical problem.
>
>
>
> What timers do people use for HSRP on BGP routers as a practice?
> Obviously we want the smallest timers that would be possible.
>
>
>
> ----
>
> Matthew Huff | 1 Manhattanville Rd
>
> Director of Operations | Purchase, NY 10577
>
> OTA Management LLC | Phone: 914-460-4039
>
> aim: matthewbhuff | Fax: 914-460-4139
>
>

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

mhuff at ox

May 9, 2012, 11:49 AM

Post #5 of 10 (861 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

Average less than 10%, peek at 40%

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff | Fax:   914-460-4139

> -----Original Message-----
> From: Mack McBride [mailto:mack.mcbride [at] viawest]
> Sent: Wednesday, May 09, 2012 2:24 PM
> To: Matthew Huff; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> What is your base CPU utilization?
>
> Mack
>
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff [at] ox]
> Sent: Wednesday, May 09, 2012 12:22 PM
> To: Mack McBride; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> That's the thing, we aren't getting any BGP events, just HSRP ones. The
> netflow analysis don't show a high bandwidth utilization, that's why I
> was looking at the BGP re-calc causing the HSRP hellos to drop.
>
>
> ----
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC | Phone: 914-460-4039
> aim: matthewbhuff | Fax:   914-460-4139
>
>
> > -----Original Message-----
> > From: Mack McBride [mailto:mack.mcbride [at] viawest]
> > Sent: Wednesday, May 09, 2012 1:39 PM
> > To: Matthew Huff; 'cisco-nsp [at] puck'
> > Subject: RE: Tuning HSRP timers for BGP routers
> >
> > We use timers 2 and 6 on multiple sets of 6500s with a relatively
> > large number of VLANs.
> > The 6500 has a much less powerful CPU and seems to handle it ok.
> > However, the 7200 is a software device so through traffic can have a
> > major effect on CPU.
> > It may be you have large high BW bursts during backups or something
> > similar.
> > You should probably do some correlation analysis before blaming the
> > BGP.
> > If you are getting that kind of BGP event on a nightly basis you
> > should probably talk to your provider.
> >
> > LR Mack McBride
> > Network Architect
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-
> > bounces [at] puck] On Behalf Of Matthew Huff
> > Sent: Wednesday, May 09, 2012 7:46 AM
> > To: 'cisco-nsp [at] puck'
> > Subject: [c-nsp] Tuning HSRP timers for BGP routers
> >
> > We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
> > using default timers for the HSRP interfaces, and we are seeing
> > nightly HSRP state changes. Not a lot, but 1-2 a night. This appears
> > to only have started recently. We are looking at logs, but I assume
> > it's due to BGP cpu exhaustion. We don't see any L2 errors on the
> VLAN
> > where the HSRP is running, so I don't think it's a physical problem.
> >
> >
> >
> > What timers do people use for HSRP on BGP routers as a practice?
> > Obviously we want the smallest timers that would be possible.
> >
> >
> >
> > ----
> >
> > Matthew Huff | 1 Manhattanville Rd
> >
> > Director of Operations | Purchase, NY 10577
> >
> > OTA Management LLC | Phone: 914-460-4039
> >
> > aim: matthewbhuff | Fax: 914-460-4139
> >
> >

Attachments:

smime.p7s (5.21 KB)

mack.mcbride at viawest

May 9, 2012, 11:50 AM

Post #6 of 10 (868 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

Are you able to determine what process is active at the peek?

Mack

-----Original Message-----
From: Matthew Huff [mailto:mhuff [at] ox]
Sent: Wednesday, May 09, 2012 12:49 PM
To: Mack McBride; 'cisco-nsp [at] puck'
Subject: RE: Tuning HSRP timers for BGP routers

Average less than 10%, peek at 40%

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff | Fax:   914-460-4139

> -----Original Message-----
> From: Mack McBride [mailto:mack.mcbride [at] viawest]
> Sent: Wednesday, May 09, 2012 2:24 PM
> To: Matthew Huff; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> What is your base CPU utilization?
>
> Mack
>
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff [at] ox]
> Sent: Wednesday, May 09, 2012 12:22 PM
> To: Mack McBride; 'cisco-nsp [at] puck'
> Subject: RE: Tuning HSRP timers for BGP routers
>
> That's the thing, we aren't getting any BGP events, just HSRP ones. The
> netflow analysis don't show a high bandwidth utilization, that's why I
> was looking at the BGP re-calc causing the HSRP hellos to drop.
>
>
> ----
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC | Phone: 914-460-4039
> aim: matthewbhuff | Fax:   914-460-4139
>
>
> > -----Original Message-----
> > From: Mack McBride [mailto:mack.mcbride [at] viawest]
> > Sent: Wednesday, May 09, 2012 1:39 PM
> > To: Matthew Huff; 'cisco-nsp [at] puck'
> > Subject: RE: Tuning HSRP timers for BGP routers
> >
> > We use timers 2 and 6 on multiple sets of 6500s with a relatively
> > large number of VLANs.
> > The 6500 has a much less powerful CPU and seems to handle it ok.
> > However, the 7200 is a software device so through traffic can have a
> > major effect on CPU.
> > It may be you have large high BW bursts during backups or something
> > similar.
> > You should probably do some correlation analysis before blaming the
> > BGP.
> > If you are getting that kind of BGP event on a nightly basis you
> > should probably talk to your provider.
> >
> > LR Mack McBride
> > Network Architect
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-
> > bounces [at] puck] On Behalf Of Matthew Huff
> > Sent: Wednesday, May 09, 2012 7:46 AM
> > To: 'cisco-nsp [at] puck'
> > Subject: [c-nsp] Tuning HSRP timers for BGP routers
> >
> > We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
> > using default timers for the HSRP interfaces, and we are seeing
> > nightly HSRP state changes. Not a lot, but 1-2 a night. This appears
> > to only have started recently. We are looking at logs, but I assume
> > it's due to BGP cpu exhaustion. We don't see any L2 errors on the
> VLAN
> > where the HSRP is running, so I don't think it's a physical problem.
> >
> >
> >
> > What timers do people use for HSRP on BGP routers as a practice?
> > Obviously we want the smallest timers that would be possible.
> >
> >
> >
> > ----
> >
> > Matthew Huff | 1 Manhattanville Rd
> >
> > Director of Operations | Purchase, NY 10577
> >
> > OTA Management LLC | Phone: 914-460-4039
> >
> > aim: matthewbhuff | Fax: 914-460-4139
> >
> >

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

tvarriale at comcast

May 9, 2012, 4:06 PM

Post #7 of 10 (866 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

On 5/9/2012 8:45 AM, Matthew Huff wrote:
> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are using
> default timers for the HSRP interfaces, and we are seeing nightly HSRP state
> changes. Not a lot, but 1-2 a night. This appears to only have started
> recently. We are looking at logs, but I assume it's due to BGP cpu
> exhaustion. We don't see any L2 errors on the VLAN where the HSRP is
> running, so I don't think it's a physical problem.
>
>
>
> What timers do people use for HSRP on BGP routers as a practice? Obviously
> we want the smallest timers that would be possible.
>
>
>
> ----
>
> Matthew Huff | 1 Manhattanville Rd
>
> Director of Operations | Purchase, NY 10577
>
> OTA Management LLC | Phone: 914-460-4039
>
> aim: matthewbhuff | Fax: 914-460-4139
>
That's the symptom. I wouldn't recommend changing those timers until
you understand what is causing the issue. Sometimes (especially in STP,
BGP, etc) changing how the protocol works makes things worse.

tv
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

alumbis at gmail

May 9, 2012, 7:38 PM

Post #8 of 10 (860 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

A problem like this, especially on a software platform like the 7200
is ususally due to either small CPU spikes, local interface issues,
transit l2 issues.

Are the problems only in the evenings? Any backups that might be
causing traffic bursts or some sort of oversubscription in the
network?

Are you using the onboard NPE ports for the connectivity between the
peers or a PA? If so, which?

Do you see any STP changes on the transit switches?

Are you seeing any other control traffic issues, say BGP, OSPF or even
HSRP on other VLANs?

The only comment I'd have on BGP exhaustion is that with default HSRP
timers, BGP convergence, even with a large event, would (hopefully)
yield the CPU to other processes to allow for at least one HSRP packet
to get processed and keep the peers up.

I think the easy thing you can do is deploy CPU threshold logging
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_cpu_thresh_notif_ps6441_TSD_Products_Configuration_Guide_Chapter.html

This will give you notification at the 5 second level.

You can get tighter with EEM by setting an event trigger on the CPU
OID and then printing out whatever relevant information (i.e. "show
proc cpu sort" | append flash:eem.txt").

-Pete

On Wed, May 9, 2012 at 9:45 AM, Matthew Huff <mhuff [at] ox> wrote:
> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are using
> default timers for the HSRP interfaces, and we are seeing nightly HSRP state
> changes. Not a lot, but 1-2 a night. This appears to only have started
> recently. We are looking at logs, but I assume it's due to BGP cpu
> exhaustion. We don't see any L2 errors on the VLAN where the HSRP is
> running, so I don't think it's a physical problem.
>
>
>
> What timers do people use for HSRP on BGP routers as a practice? Obviously
> we want the smallest timers that would be possible.
>
>
>
> ----
>
> Matthew Huff | 1 Manhattanville Rd
>
> Director of Operations | Purchase, NY 10577
>
> OTA Management LLC | Phone: 914-460-4039
>
> aim: matthewbhuff | Fax: 914-460-4139
>
>
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

mitch at stonernetworks

May 9, 2012, 8:28 PM

Post #9 of 10 (837 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

Perhaps EEM is something that could be helpful here...

> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
> using
> default timers for the HSRP interfaces, and we are seeing nightly HSRP
> state
> changes. Not a lot, but 1-2 a night. This appears to only have started
> recently. We are looking at logs, but I assume it's due to BGP cpu
> exhaustion. We don't see any L2 errors on the VLAN where the HSRP is
> running, so I don't think it's a physical problem.
>
>
>
> What timers do people use for HSRP on BGP routers as a practice?
> Obviously
> we want the smallest timers that would be possible.
>
>
>
> ----
>
> Matthew Huff | 1 Manhattanville Rd
>
> Director of Operations | Purchase, NY 10577
>
> OTA Management LLC | Phone: 914-460-4039
>
> aim: matthewbhuff | Fax: 914-460-4139
>
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

alumbis at gmail

May 11, 2012, 5:05 AM

Post #10 of 10 (838 views)
Permalink

Re: Tuning HSRP timers for BGP routers [In reply to]

Mitch,

You can deploy something like this

===========================
event manager applet cpuspike
event snmp oid "1.3.6.1.4.1.9.9.109.1.1.1.1.10.1" get-type next
entry-op ge entry-val "50" exit-time 60 poll-interval 1
action 1.0 syslog msg "CPU Spike Detected - Please issue more
flash:cpuinfo for data"
action 2.0 cli command "en"
action 3.0 cli command "show proc cpu sort | append flash:cpuinfo"
action 4.0 cli command "show proc cpu history | append flash:cpuinfo"
action 5.0 cli command "show interface | append flash:cpuinfo"
===========================

This will poll on a 1 second interval and trigger anytime CPU is over
50 percent. After it fires it will sleep for 60 seconds before
attempting to run a second time.

Since this requires enable mode you may need to add "event manager
session cli username <username>" to the global config where the
username will be someone with the authorization to run the commands
listed.

I'd suggest double checking this OID as I think this is from a script
I used on a 6k, not for the 7200. Make sure you are using the current
CPU OID and not the 5 second CPU OID.

If you want to test it just change the "entry-val 50" to say, 1 and
run a "show tech" or anything else that would cause a small bump in
CPU and you should see it trigger.

-Pete

On Wed, May 9, 2012 at 11:28 PM, <mitch [at] stonernetworks> wrote:
> Perhaps EEM is something that could be helpful here...
>
>> We have a pair of Cisco 7204VXR with NPE-G2 running 15.1(4)M3. We are
>> using
>> default timers for the HSRP interfaces, and we are seeing nightly HSRP
>> state
>> changes. Not a lot, but 1-2 a night. This appears to only have started
>> recently. We are looking at logs, but I assume it's due to BGP cpu
>> exhaustion. We don't see any L2 errors on the VLAN where the HSRP is
>> running, so I don't think it's a physical problem.
>>
>>
>>
>> What timers do people use for HSRP on BGP routers as a practice?
>> Obviously
>> we want the smallest timers that would be possible.
>>
>>
>>
>> ----
>>
>> Matthew Huff | 1 Manhattanville Rd
>>
>> Director of Operations | Purchase, NY 10577
>>
>> OTA Management LLC | Phone: 914-460-4039
>>
>> aim: matthewbhuff | Fax: 914-460-4139
>>
>>
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp [at] puck
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads